这篇笔记比较老,不再更新维护,请移步最新的手册:envoy相关笔记。
@2019-01-17 15:16:31 IDE建议使用vscode
,直接跳转到章节:在Mac上准备Envoy的开发环境。
之前用Clion折腾了很久,发现Clion不能正确解读用bazel-cmakelists生成的CMakeLists.txt。后面关于Clion的内容先保留着,还是有一定价值的。
使用C/C++开发使用的Clion阅读envoy代码,Clion使用cmake管理构建项目的,envoy使用的是bazel。 Clion中可以安装一个名为bazel的插件的,不知为何安装之后没有效果,软件界面也和网上的教程不匹配。
这种方式不能索引envoy依赖的外部代码! 建议直接参照第三节中的方法,用vscode + bazel-cmakelists。
如果只是用Clion阅读、编写Envoy代码,将Envoy代码导入后,需要将envoy项目目录中的自动生成的CMakeLists.txt
文件内容修改为如下:
cmake_minimum_required(VERSION 3.13)
project(envoy)
set(CMAKE_CXX_STANDARD 14)
include_directories(include)
include_directories(source)
# add_executable保持原状
add_executable(envoy
api/test/build/build_test.cc
api/test/validate/pgv_test.cc
..省略...)
include目录只需要包含include
和source
目录就可以了,因为envoy的代码中引用头文件的时候,使用的是相对于include和source的路径,例如:
#include "exe/main_common.h" // 引用的是:source/exe/main_common.h
#include "absl/debugging/symbolize.h" // 部分找不到头文件是envoy依赖的外部库
这一步比较折腾,如果不在意bazel-cmakelists的用法,可以直接使用下一节中的操作,相关文件已经设置好了。
如果要在Clion中编译envoy,并且希望能够看到envoy依赖的代码,可以尝试一下Github上一个名为bazel-cmakelists的工具, 这个工具可以为使用Bazel的C++项目生成CMakeLists.txt。
本地需要安装有bazel,mac上可用brew安装bazel:
brew tap bazelbuild/tap
brew tap-pin bazelbuild/tap
brew install bazelbuild/tap/bazel
下载bazel-cmakelists,将其放在envoy项目根目录中:
cd envoy
git clone https://github.com/lizan/bazel-cmakelists.git
bazel-cmakelists
命令执行过程中会调用bazel命令构建指定的target,如果bazel build //source/exe:envoy-static
能够运行,可以直接在envoy项目目录中执行下面的命令:
./bazel-cmakelists/bazel-cmakelists --targets //source/exe:envoy-static
如果是在容器中构建envoy,转换过程比较麻烦。
需要在ci/do_ci.sh中增加一个名为cmake
的任务:
elif [[ "$1" == "cmake" ]]; then
setup_gcc_toolchain
echo "bazel cmake..."
cd "${ENVOY_CI_DIR}"
../bazel-cmakelists/bazel-cmakelists --targets //source/exe:envoy-static --output=../CMakeLists.txt
exit 0
修改./bazel-cmakelists/bazel-cmakelists
:
# 将下面这行
bazel_args=['build']
# 修改为
bazel_args=['build', '--strategy=Genrule=standalone', '--spawn_strategy=standalone', '--verbose_failures', '--package_path', '%workspace%:/source', '--action_env=HOME', '--action_env=PYTHONUSERBASE', '--jobs=4', '--show_task_finish', '-c', 'opt']
还需要将./bazel-cmakelists/bazel-cmakelists
中所有用到bazel命令的地方加上参数'--package_path', '%workspace%:/source'
,例如:
for fn in subprocess.check_output(['bazel', 'query', '--package_path', '%workspace%:/source', '--noimplicit_deps', query]).splitlines():
xml = subprocess.check_output(['bazel', 'query','--package_path', '%workspace%:/source', '--noimplicit_deps', query, '--output', 'xml'])
...
然后执行下面的命令:
$ ./ci/run_envoy_docker.sh './ci/do_ci.sh cmake'
building using 4 CPUs
gcc/g++ toolchain configured
bazel cmake...
$TEST_TMPDIR defined: output root default is '/build/tmp' and max_idle_secs default is '15'.
Starting local Bazel server and connecting to it...
$TEST_TMPDIR defined: output root default is '/build/tmp' and max_idle_secs default is '15'.
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
currently loading: source/exe
Analyzing: target //source/exe:envoy-static (3 packages loaded)
Analyzing: target //source/exe:envoy-static (9 packages loaded)
Analyzing: target //source/exe:envoy-static (84 packages loaded)
Analyzing: target //source/exe:envoy-static (176 packages loaded)
Analyzing: target //source/exe:envoy-static (187 packages loaded)
Analyzing: target //source/exe:envoy-static (259 packages loaded)
Analyzing: target //source/exe:envoy-static (269 packages loaded)
Analyzing: target //source/exe:envoy-static (285 packages loaded)
INFO: Analysed target //source/exe:envoy-static (290 packages loaded).
INFO: Found 1 target...
[1 / 7] [-----] BazelWorkspaceStatusAction stable-status.txt
...
得到CMakeLists.txt
以后,还需要修正其中的路径,因为这个文件是在容器生成的,文件中的路径是容器中的路径,如果与容器外路径不一致,需要修改为容器外的路径,例如:
set(INCLUDE_DIRECTORIES
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/external/com_github_google_jwt_verify
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/external/com_github_grpc_grpc/third_party/address_sorting
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/bazel-out/k8-opt/genfiles/external/com_github_gabime_spdlog/include
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/bazel-out/k8-opt/genfiles/external/com_github_eile_tclap/include
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/bazel-out/k8-opt/genfiles/external/com_lyft_protoc_gen_validate
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/external/bazel_tools
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/external/com_github_gabime_spdlog/include
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/external/boringssl/src/include
/build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/bazel-out/k8-opt/genfiles/external/boringssl/src/includ
这里的/build是容器中的路径,容器外的路径是用变量ENVOY_DOCKER_BUILD_DIR
指定的,默认是tmp
。
建议使用vscode,用cilon的时候,即使用baze-cmakelists生成了CMakeLists.txt,cilon还是找不到头文件。 vscode就没有这个问题,按照后面的操作生成CMakeLists.txt之后,直接将eonvy目录导入vscode就可以了,非常好用,envoy自身的头文件和依赖的外部头文件都能找到。
安装vscode,直接到vscode的网站上下载。
安装bazel:
brew tap bazelbuild/tap
brew tap-pin bazelbuild/tap
brew install bazelbuild/tap/bazel
下载envoy代码,这里切换到v1.8.0版本中:
git clone https://github.com/envoyproxy/envoy.git
cd envoy
git checkout v1.8.0
在envoy目录中
下载经过修改的bazel-cmakelists
:
cd envoy
git clone https://github.com/lijiaocn/envoy-bazel-cmakelists.git
在ci/do_ci.sh
中加入一个新的任务:
elif [[ "$1" == "cmake" ]]; then
setup_gcc_toolchain
echo "bazel cmake..."
cd "${ENVOY_CI_DIR}"
../envoy-bazel-cmakelists/bazel-cmakelists --targets //source/exe:envoy-static --output=../CMakeLists.txt
exit 0
在本地准备bazel的缓存目录,这里指定为/build
,与docker中路径相同,这样就不需要调整生成的CMakeLists.txt文件了:
export ENVOY_DOCKER_BUILD_DIR=/build
sudo mkdir $ENVOY_DOCKER_BUILD_DIR
sudo chown -R lijiao $ENVOY_DOCKER_BUILD_DIR
如果不想每次都设置环境变量,可以在脚本ci/run_envoy_docker.sh
开始的地方直接定义变量:
ENVOY_DOCKER_BUILD_DIR=/build
在Mac上还需要设置一下docker,在docker的Perferences->File Sharing
面板中添加刚创建的/build
目录,这一步一定要有,否则会报下面的错误:
➜ envoy git:(5d25f466c) ✗ ./ci/run_envoy_docker.sh './ci/do_ci.sh cmake'
docker: Error response from daemon: Mounts denied:
The path /build
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
在docker的配置面板中添加/build目录之后,需要点击配置面板中的Apply&Start
才能生效。
用下面的命令生成CMakeLists.txt文件:
./ci/run_envoy_docker.sh './ci/do_ci.sh cmake'
生成CMakeLists.txt之前,会调用bazel编译envoy,envoy第一次编译的时间非常非常长,4核CPU的情况下都要几个小时。
编译过程中可能遇到下面的问题:
$TEST_TMPDIR defined: output root default is '/build/tmp' and max_idle_secs default is '15'.
Starting local Bazel server and connecting to it...
ERROR: /source/bazel/repositories.bzl:315:5: no such package '@boringssl//': BUILD file not found on package path and referenced by '//external:ssl'
ERROR: Analysis of target '//source/exe:envoy-static' failed; build aborted: Analysis failed
INFO: Elapsed time: 8.258s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (179 packages loaded)
currently loading: @com_google_protobuf//
rm: cannot remove '/source/bazel-cmakelists': Is a directory
出现这个问题的原因不明确,有时候这一次运行和下一次运行的错误还不相同,怀疑是bazel在运行时要用git拉取很多代码,但是网速速度比较慢,有些代码拉取失败,导致后续一些列错误。 将/build目录中的文件清除后,在网络状况好的时候重试就可以了。
正常情况应当出现下面的日志:
Analyzing: target //source/exe:envoy-static (266 packages loaded)
INFO: SHA256 (https://github.com/prometheus/client_model/archive/99fa1f4be8e564e8a6b613da7fa6f46c9edafc6c.tar.gz) = 783bdaf8ee0464b35ec0c8704871e1e72afa0005c3f3587f65d9d6694bf3911b
INFO: Analysed target //source/exe:envoy-static (290 packages loaded).
INFO: Found 1 target...
[62 / 280] 4 actions running
GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg [for host]; 72s local
Compiling external/com_google_protobuf/src/google/protobuf/message.cc [for host]; 4s local
Compiling external/com_google_protobuf/src/google/protobuf/compiler/zip_writer.cc [for host]; 1s local
Compiling external/com_google_protobuf/src/google/protobuf/util/field_mask_util.cc [for host]; 0s local
还经常遇到下面的问题Server crashed during startup
,原因同样不明,重新执行就好了,似乎某部外部条件导致bazel crash:
No remote cache bucket is set, skipping setup remote cache.
ENVOY_SRCDIR=/source
find: File system loop detected; '/source/ci/bazel-ci/ci' is part of the same file system loop as '/source/ci'.
find: File system loop detected; '/source/ci/bazel-ci/external/envoy/ci' is part of the same file system loop as '/source/ci'.
HEAD is now at 3e5b733... cleanup: match NOT_IMPLEMENTED_GCOVR_EXCL_LINE change (#53)
building using 4 CPUs
gcc/g++ toolchain configured
bazel release build...
Building...
$TEST_TMPDIR defined: output root default is '/build/tmp' and max_idle_secs default is '15'.
Starting local Bazel server and connecting to it...
Server crashed during startup. Now printing /build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/server/jvm.out
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/build/tmp/_bazel_bazel/install/ca1b8b7c3e5200be14b7f27896826862/_embedded_binaries/A-server.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
如果使用CentOS镜像编译,有可能遇到下面的问题:
In file included from source/exe/main_common.cc:6:0:
bazel-out/k8-opt/bin/source/common/common/_virtual_includes/compiler_requirements_lib/common/common/compiler_requirements.h:14:2: error: #error "Your toolchain has set _GLIBCXX_USE_CXX11_ABI to a value that uses a std::string " "implementation that is not thread-safe. This may cause rare and difficult-to-debug errors " "if std::string is passed between threads in any way. If you accept this risk, you may define " "ENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR=1 in your build."
#error "Your toolchain has set _GLIBCXX_USE_CXX11_ABI to a value that uses a std::string "
研究了bazel半天,没找到添加预定义宏的地方,后来同事找到了一个临时方法,直接把代码修改了:
将source/common/common/compiler_requirements.h
中第13行前面的!
去掉:
#if defined(_GLIBCXX_USE_CXX11_ABI) && _GLIBCXX_USE_CXX11_ABI != 1 && \
!defined(ENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR)
修改成:
#if defined(_GLIBCXX_USE_CXX11_ABI) && _GLIBCXX_USE_CXX11_ABI != 1 && \
defined(ENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR)
编译完成,envoy目录中生成了CMakeLists.txt
文件之后,直接将envoy目录导入到vscode中,就可以愉快的读代码、写代码了。
如果还是提示头文件找不到,编辑.vscode目录的中的json文件,在includePath中添加上/build/**
:
"includePath": [
"${workspaceFolder}/**",
"/build/**"
],
然后还可以安装Bazel Tools
插件。
Find all references中讨论了很久,没耐心看他们在讨论些什么…
尝试了一下讨论中提到的cquery插件,安装后发现可用。
安装cquery,在mac上可以直接用brew安装:
brew install cquery
cquery和vscode官方的c++插件冲突,后来发现C++ Intellisense更好一些。
在vscode中安装了C++ Intellisense
插件之后,还需要在本地安装gnu global tool,在mac上可以直接用brew安装:
brew install global
然后到项目顶层目录中执行命令gtags
生成tag文件。