Build impossible on Centos 7

changed the description

Isn't this the same problem with spdlog linking that we solved a while ago for CentOS?

It is clear: we have to run a CentOS build on CI. This is too fragile.

Any experts recalls this particular problem and how to best solve it?

Indeed, there was this problem. Maybe spdlog was with a different version.

I'll have to try on CentOS 7 and also prepare a CI container for CentOS 7. I don't recall the solution when this came up first.

The problem was mainly these old/new ABI.

I have just tried to build again corsika, current master, on my centos7 machine, and it looks like the spdlog issue is back ?

[ 75%] Building CXX object tests/framework/CMakeFiles/testFramework.dir/testSolver.cpp.o [ 75%] Linking CXX executable ../../bin/testFramework /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /home/bregeon/.conan/data/boost/1.76.0/_/_/package/ca891ed5c8e67ebb3e46639c100abdbca6d8b11e/lib/libboost_filesystem.a(operations.o): in function boost::filesystem::detail::copy_file(boost::filesystem::path const&, boost::filesystem::path const&, unsigned int, boost::system::error_code*)': operations.cpp:(.text+0x29b): undefined reference to statx' /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: operations.cpp:(.text+0x44f): undefined reference to statx' /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /home/bregeon/.conan/data/boost/1.76.0///package/ca891ed5c8e67ebb3e46639c100abdbca6d8b11e/lib/libboost_filesystem.a(operations.o): in function boost::filesystem::detail::copy_directory(boost::filesystem::path const&, boost::filesystem::path const&, boost::system::error_code*)': operations.cpp:(.text+0x60d): undefined reference to statx'

(...)

/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/testFramework.dir/testCascade.cpp.o: in function void spdlog::logger::log_<fmt::v7::basic_string_view<char>, int>(spdlog::source_loc, spdlog::level::level_enum, fmt::v7::basic_string_view<char> const&, int&&)': /home/bregeon/.conan/data/spdlog/1.8.5/_/_/package/a8b001c04e44ed5cb4208ab0f8bf279c6689b496/include/spdlog/logger.h:344: undefined reference to spdlog::logger::err_handler_(std::string const&)' /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/testFramework.dir/testCascade.cpp.o:/home/bregeon/.conan/data/spdlog/1.8.5///package/a8b001c04e44ed5cb4208ab0f8bf279c6689b496/include/spdlog/logger.h:344: more undefined references to `spdlog::logger::err_handler_(std::string const&)' follow collect2: error: ld returned 1 exit status make[2]: *** [tests/framework/CMakeFiles/testFramework.dir/build.make:472: bin/testFramework] Error 1 make[1]: *** [CMakeFiles/Makefile2:1021: tests/framework/CMakeFiles/testFramework.dir/all] Error 2 make: *** [Makefile:161: all] Error 2

`

can anyone confirm?

Besides, I also had to work around a cmake "version" issue, conan looking for cmake internally instead of using cmake3 (had to make a simlink).

Reporting here the email thread from 28/02/2021:

... and I found a way to fix the problem: make a non-const string and pass it to setting the pattern. I'll make a branch now to deal with the problem.

Lukas

(conan) [lukas@centos7 build]$ export CORSIKA_DATA=~/CORSIKA8/corsika/modules/data
(conan) [lukas@centos7 build]$ ctest3
Test project /home/lukas/CORSIKA8/build
    Start 1: copyright_notices
1/6 Test #1: copyright_notices ................   Passed    0.14 sec
    Start 2: testData
2/6 Test #2: testData .........................   Passed    0.01 sec
    Start 3: testFramework
3/6 Test #3: testFramework ....................   Passed    2.26 sec
    Start 4: testMedia
4/6 Test #4: testMedia ........................   Passed    0.02 sec
    Start 5: testStack
5/6 Test #5: testStack ........................   Passed    0.01 sec
    Start 6: testModules
6/6 Test #6: testModules ......................   Passed  130.18 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 132.68 sec
(conan) [lukas@centos7 build]$ gcc --version
gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

On 28/02/21 14:25 , Lukas Nellen wrote:
> Hi Ralf,
>
> In the case which I ran with gdb, the problem occurs when corsika sets up the default pattern for the logger:
>
>     /home/lukas/CORSIKA8/corsika/corsika/detail/framework/core/Logging.inl:50
>         logger->set_pattern(logging::default_pattern);
>
> As far as I can see from the code, this would involve a move constructor or move assign for the string representing the default pattern. The part of the stack trace in the system libraries suggests a memory leak:
>
>     #0  std::string::_Rep::_M_is_leaked (this=0xffffffffffffffe8)
>         at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3190
>     #1  0x000000000045751c in std::string::_Rep::_M_grab (this=0xffffffffffffffe8, __alloc1=..., __alloc2=...)
>         at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3241
>     #2  0x000000000044934a in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (
>         this=0x7fffffffddf8, __str=<error reading variable: Cannot access memory at address 0xffffffffffffffe8>)
>         at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.tcc:613
>     #3  0x00000000004cd20d in corsika::create_logger (name="corsika", defaultlog=true)
>         at /home/lukas/CORSIKA8/corsika/corsika/detail/framework/core/Logging.inl:50
>
> which makes me suspect a problem with moving objects, since that is a new operation in C++ 11. I therefore suspect that the old, pre C++ implementation of string, used in the old ABI, doesn't implement the necessary operations correctly.
>
>     In the GCC 5.1 release libstdc++ introduced a new library ABI that includes new implementations of std::string and std::list.
>
> The compiler (and accompanying libstdc++) on CentOS is 4.8.5, i.e., it predates the ABI changes. The devtoolset compilers from the software collection are built to maintain run-time compatibility so the compiled code can run also on machines w/o the devtoolset installed. To achieve this, they use the libraries in the core CentOS and not new libraries. Which is the longer explanation why we have to build with the old ABI on CentOS 7.
>
> So what all this means: I don't think the segfault is directly linked to spdlog, it just happens to include code that triggers the bug. The problem happens even when I use the header-only version of spdlog, which contains no code build on a different platform.
>
> Cheers,
> Lukas
>
> On 28/02/21 13:24 , Ralf Ulrich wrote:
>> Hi Lukas,
>>
>> this sounds like great progress. Careful about the segfault:    this may be a normal behavior....  The testModules always segfault, if CORSIKA_DATA  variable is not properly set.  Have you set CORSIKA_DATA to "<your_corsika>/modules/data" ? This should be changed, but has not been done so far.
>>
>> Or is the segfault really linked to spdlog?
>>
>> Hm, about the general problem with the ABI. Isn't this one of the things that conan should be able to fix for us? We can't be the only group affected by this.
>>
>> Cheers,
>> Ralf
>>
>> On 27.02.21 05:19, Lukas Nellen wrote:
>>> I finally had some time to look int this issue.
>>>
>>> The problem arises because of the C++ abi used by default. I can compile code when I force conan to pull packages build with the old abi:
>>>     cmake ../corsika/ -D_GLIBCXX_USE_CXX11_ABI=OFF
>>> This way, the configuration
>>>     compiler.libcxx=libstdc++
>>> gets used instead of
>>>     compiler.libcxx=libstdc++11
>>>
>>> The code builds, but I get segmentation faults when running the tests. I'm not sure what happens there. It could be a problem in the internals of the sdplog binary or a problem of the old libstdc++ used by CentOS 7. Could be moving an object, which is new in c++11 and probably has no support in the old libstdc++. At least the context of the bug, looking a a stack trace in gdb, suggests that c++ would move a string there.
>>>
>>> The problem occurs with devtoolsets 7, 8, and 9. I tried to pull in the header-only version of spdlog adding
>>>
>>>     [options]
>>>     spdlog:header_only=True
>>>     fmt:header_only=True
>>>
>>> to the conanfile.txt. But it doesn't seem to help.
>>>
>>> Things (more or less) work If I use the new ABI and a compiler that doesn't try to remain compatible with the system libraries on CentOS 7. I tried with gcc 8.3.0.1 from
>>>     /cvmfs/sft.cern.ch/lcg/contrib/gcc/8.3.0.1/x86_64-centos7
>>> which was the easiest to try, since I have access to cvmfs on my centos 7 box.
>>>
>>> I get a segfault in testModules both on my debian box and on centos 7. Not sure if I did something wrong with the submodules - the fault happens testing qgsjet II
>>>
>>> Cheers,
>>> Lukas
>>>

Thanks, unfortunately I am not even surprised....

If we want this to become stable on CentOS, we MUST have a CentOS CI Runner to replicate the current problem and we sensitive to the fix.

All other fixed will remain temporary and subject to future fast deterioration.

Who is most interested to ultimately fix this issue? @lukas can we provide such a centos container? Or do we have it already?

added Continuous Integration label

mentioned in issue #566 (closed)

Build impossible on Centos 7

Child items ...

Activity