Zipkin Dubbo安装配置使用

zipkin
dubbo, zipkin

前言

生产环境上的几个由dubbo发布的微服务，支付，订单系统模块，当测试反馈或者用户反馈相应慢的时候，排查到底是在哪个节点服务出问题时，这可就麻烦了，这里牵涉到整个链路跟踪。当时也用过傻办法，每个方法出入口都有对应的消耗时间然后痛苦的去查看日志排查。这里记录下使用dubbo整合zipkin来跟踪整个服务链路消耗时间。

准备安装文件

以下为大家准备了安装文件，也可以自己到网上下载

下载地址：curl -sSL https://zipkin.io/quickstart.sh | bash -s

准备文件 下载地址
zipkin.jar 链接: https://pan.baidu.com/s/1dlUjcLjemTcm7jnc0HDiwQ 提取码: typ4

Zipkin安装启动

由于是一个jar文件，直接下载就可以通过java -jar方式允许（前提先安装java环境）

以上为zipkin的安装启动，还是比较简单的。不过按照这种方式安装的 Zipkin Server 使用的存储类型是 inMemory 的。当服务器停机之后，所有收集到的 trace 信息会丢失，不适用于生产系统。如果在生产系统中使用，需要配置另外的存储类型。Zipkin 支持 MySql、Cassandra、和 ElasticSearch。推荐使用 Cassandra 和 ElasticSearch。后续上生产后，在记录下对应的配置，这里也仅仅是先把整个流程跑通。接下来就是需要对需要收集的项目进行简单改造，增加配置信息。

Dubbo项目ZipKin配置

  • 增加pom配置依赖(全局依赖)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    <brave.version>4.19.2</brave.version>
    <zipkin-reporter.version>2.1.3</zipkin-reporter.version>
    <zipkin.version>2.8.1</zipkin.version>

    <dependency>
        <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-reporter</artifactId>
        <version>2.6.0</version>
        <type>pom</type>
    </dependency>
    <dependency>
        <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-sender-okhttp3</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-context-log4j2</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-instrumentation-http</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-instrumentation-http-tests</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-instrumentation-servlet</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-tests</artifactId>
        <version>4.19.2</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.zipkin2</groupId>
        <artifactId>zipkin</artifactId>
        <version>${zipkin.version}</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.java</groupId>
        <artifactId>zipkin</artifactId>
        <version>${zipkin.version}</version>
    </dependency>
    <dependency>
        <groupId>io.zipkin.java</groupId>
        <artifactId>zipkin-junit</artifactId>
        <version>${zipkin.version}</version>
    </dependency>

  • 增加zipkin公共模块config，以及对应的filter代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    package com.smarttrip.pay.common.zipkin;

    import brave.Tracing;
    import brave.servlet.TracingFilter;
    import org.springframework.beans.factory.annotation.Value;
    import org.springframework.boot.web.servlet.FilterRegistrationBean;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import zipkin2.Span;
    import zipkin2.reporter.AsyncReporter;
    import zipkin2.reporter.Reporter;
    import zipkin2.reporter.Sender;
    import zipkin2.reporter.okhttp3.OkHttpSender;

    import javax.servlet.Filter;

    /**
     * @program: 配置文件
     * @description:
     * @author: lishijia
     * @create: 2019-01-23 10:31
     **/
    @Configuration
    public class TracingConfig {

        /**
         * 配置zipkin服务地址
         */
        @Value("${zipkin.tracing.endpoint:endpoint}")
        private String zipkinEndPoint;

        @Value("${zipkin.tracing.local-service-name:local-service-name}")
        private String localServiceName;

        /**
         * 配置sender
         * @return
         */
        @Bean
        public Sender sender(){
            OkHttpSender sender = OkHttpSender
                    .newBuilder()
                    .endpoint(zipkinEndPoint)
                    .build();
            return sender;
        }

        /**
         * 配置reporter
         * @param sender
         * @return
         */
        @Bean
        public Reporter<Span> reporter(Sender sender){
            return AsyncReporter
                    .builder(sender)
                    .build();
        }

        /**
         * 配置dubbo-consumer tracing
         * @param reporter
         * @return
         */
        @Bean
        public Tracing tracing(Reporter reporter){
            return Tracing.newBuilder()
                    .localServiceName(localServiceName)
                    .spanReporter(reporter)
                    .build();
        }

        /**
         * 配置http tracing
         * @param reporter
         * @return
         */
        @Bean
        public Tracing tracing2(Reporter reporter){
            return Tracing.newBuilder()
                    .localServiceName(localServiceName + "_http")
                    .spanReporter(reporter)
                    .build();
        }

        /**
         * 配置servlet filter
         * @param tracing2
         * @return
         */
        @Bean
        public Filter filter(Tracing tracing2){
            return TracingFilter.create(tracing2);
        }

        /**
         * 注册filter
         * @param filter
         * @return
         */
        @Bean
        public FilterRegistrationBean filterRegistration(Filter filter) {
            FilterRegistrationBean registration = new FilterRegistrationBean();
            registration.setFilter(filter);
            registration.addUrlPatterns("/*");
            registration.setName("zipkin-filter");
            registration.setOrder(1);
            return registration;
        }
    }
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    package com.smarttrip.pay.common.zipkin;


    import brave.Span;
    import brave.Tracer;
    import brave.propagation.Propagation;
    import com.alibaba.dubbo.remoting.exchange.ResponseCallback;
    import com.alibaba.dubbo.rpc.*;
    import com.alibaba.dubbo.rpc.protocol.dubbo.FutureAdapter;
    import com.alibaba.dubbo.rpc.support.RpcUtils;
    import com.alibaba.fastjson.JSON;
    import zipkin2.Endpoint;

    import java.net.InetSocketAddress;
    import java.util.Map;
    import java.util.concurrent.Future;

    /**
     * @program:
     * @description:
     * @author: lishijia
     * @create: 2019-01-23 10:31
     **/
    public class ZipkinHelper {

        static final Propagation.Setter<Map<String, String>, String> SETTER =
                new Propagation.Setter<Map<String, String>, String>() {
                    @Override
                    public void put(Map<String, String> carrier, String key, String value) {
                        carrier.put(key, value);
                    }
                    @Override
                    public String toString() {
                        return JSON.toJSONString(this);
                    }
                };
        static final Propagation.Getter<Map<String, String>, String> GETTER =
                new Propagation.Getter<Map<String, String>, String>() {
                    @Override
                    public String get(Map<String, String> carrier, String key) {
                        return carrier.get(key);
                    }

                    @Override
                    public String toString() {
                        return JSON.toJSONString(this);
                    }
                };

        static void buildSpan(Span span, Span.Kind kind, InetSocketAddress remoteAddress, String service, String method){
            if (!span.isNoop()) {
                span.kind(kind).start();
                span.kind(kind);
                span.name(service + "/" + method);
                Endpoint.Builder remoteEndpoint = Endpoint.newBuilder().port(remoteAddress.getPort());
                if (!remoteEndpoint.parseIp(remoteAddress.getAddress())) {
                    remoteEndpoint.parseIp(remoteAddress.getHostName());
                }
                span.remoteEndpoint(remoteEndpoint.build());
            }
        }

        static Result spanTracing(Span span, Tracer tracer, Invoker<?> invoker, Invocation invocation, RpcContext rpcContext){
            boolean isOneway = false;
            boolean deferFinish = false;
            try (Tracer.SpanInScope scope = tracer.withSpanInScope(span)) {
                Result result = invoker.invoke(invocation);
                if (result.hasException()) {
                    onError(result.getException(), span);
                }
                isOneway = RpcUtils.isOneway(invoker.getUrl(), invocation);
                Future<Object> future = rpcContext.getFuture(); // the case on async client invocation
                if (future instanceof FutureAdapter) {
                    deferFinish = true;
                    ((FutureAdapter) future).getFuture().setCallback(new ZipkinHelper.FinishSpanCallback(span));
                }
                return result;
            } catch (Exception e) {
                onError(e, span);
                throw e;
            } finally {
                if (isOneway) {
                    span.flush();
                } else if (!deferFinish) {
                    span.finish();
                }
            }
        }

        static void onError(Throwable error, Span span) {
            span.error(error);
            if (error instanceof RpcException) {
                span.tag("dubbo.error_code", Integer.toString(((RpcException) error).getCode()));
            }
        }

        static final class FinishSpanCallback implements ResponseCallback {
            final Span span;

            FinishSpanCallback(Span span) {
                this.span = span;
            }

            @Override public void done(Object response) {
                span.finish();
            }

            @Override public void caught(Throwable exception) {
                onError(exception, span);
                span.finish();
            }
        }
    }
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    package com.smarttrip.pay.common.zipkin;

    /**
     * @program: DubboZipkinProviderFilter是dubbo服务端过滤器,用于拦截zipkin上下文消息
     * @description:
     * @author: lishijia
     * @create: 2019-01-2310:33
     **/
    import brave.Span;
    import brave.Tracer;
    import brave.Tracing;
    import brave.propagation.TraceContext;
    import brave.propagation.TraceContextOrSamplingFlags;
    import com.alibaba.dubbo.common.Constants;
    import com.alibaba.dubbo.common.extension.Activate;
    import com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory;
    import com.alibaba.dubbo.rpc.*;
    import com.alibaba.dubbo.rpc.support.RpcUtils;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;

    import java.util.Map;

    @Activate(group = Constants.PROVIDER)
    public class DubboZipkinProviderFilter implements Filter {

        private static final Logger log = LoggerFactory.getLogger(DubboZipkinProviderFilter.class);

        private SpringExtensionFactory springExtensionFactory = new SpringExtensionFactory();
        private Tracer tracer;

        // tracing上下文消息提取
        private TraceContext.Extractor<Map<String, String>> extractor;

        @Override
        public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
            log.info("dubbo zipkin provider filter......");

            Tracing tracing = springExtensionFactory.getExtension(Tracing.class, "tracing");
            tracer = tracing.tracer();
            if (null == tracer){
                return invoker.invoke(invocation);
            }

            if (null == extractor){
                extractor = tracing.propagation().extractor(ZipkinHelper.GETTER);
            }

            TraceContextOrSamplingFlags extracted = extractor.extract(invocation.getAttachments());
            Span span = extracted.context() != null
                    ? tracer.joinSpan(extracted.context())
                    : tracer.nextSpan(extracted);

            RpcContext rpcContext = RpcContext.getContext();
            ZipkinHelper.buildSpan(span, Span.Kind.SERVER, rpcContext.getRemoteAddress(), invoker.getInterface().getSimpleName(),
                    RpcUtils.getMethodName(invocation));

            return ZipkinHelper.spanTracing(span, tracer, invoker, invocation, rpcContext);
        }
    }
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    package com.smarttrip.pay.common.zipkin;

    import brave.Span;
    import brave.Tracer;
    import brave.Tracing;
    import brave.propagation.TraceContext;
    import com.alibaba.dubbo.common.Constants;
    import com.alibaba.dubbo.common.extension.Activate;
    import com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory;
    import com.alibaba.dubbo.rpc.*;
    import com.alibaba.dubbo.rpc.support.RpcUtils;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;

    import java.util.Map;

    /**
     * @program: DubboZipkinConsumerFilter是dubbo消费端过滤器,用于拦截zipkin上下文消息
     * @description:
     * @author: lishijia
     * @create: 2019-01-23 10:32
     **/
    @Activate(group = Constants.CONSUMER)
    public class DubboZipkinConsumerFilter implements Filter {
        private static final Logger log = LoggerFactory.getLogger(DubboZipkinConsumerFilter.class);

        private SpringExtensionFactory springExtensionFactory = new SpringExtensionFactory();
        private Tracer tracer;

        // tracing上下文消息注入
        private TraceContext.Injector<Map<String, String>> injector;

        @Override
        public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
            log.info("dubbo zipkin consumer filter......");

            Tracing tracing = springExtensionFactory.getExtension(Tracing.class, "tracing");
            tracer = tracing.tracer();
            if (tracer == null){
                return invoker.invoke(invocation);
            }

            if (null == injector){
                injector = tracing.propagation().injector(ZipkinHelper.SETTER);
            }

            RpcContext rpcContext = RpcContext.getContext();
            Span span = tracer.nextSpan();
            injector.inject(span.context(), invocation.getAttachments());

            ZipkinHelper.buildSpan(span, Span.Kind.CONSUMER, rpcContext.getRemoteAddress(), invoker.getInterface().getSimpleName(),
                    RpcUtils.getMethodName(invocation));

            return ZipkinHelper.spanTracing(span, tracer, invoker, invocation, rpcContext);
        }

    }

  • 配置文件

    以下配置文件在每个服务端，消费端都需要添加

    1
    2
    3
    4
    5
    # 启用自动配置 
    # 在resources目录新建META-INF目录: 
    # 新建文件:spring.factories 内容如下：
    org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
      com.smarttrip.pay.common.zipkin.TracingConfig
    1
    2
    3
    4
    5
    # 在provider，consumer启动工程下增加如下配置过滤器：
    # 在`META-INF`添加`dubbo`文件夹, 
    # 新建文件:`com.alibaba.dubbo.rpc.Filter` 内容如下：
    zipkinConsumerDubboFilter=com.smarttrip.pay.common.zipkin.DubboZipkinConsumerFilter
    zipkinProviderDubboFilter=com.smarttrip.pay.common.zipkin.DubboZipkinProviderFilter

  • 配置每个节点的名称以及zipkin上报地址

    1
    2
    3
    4
    zipkin:
      tracing:
        local-service-name: zipkin-provider
        endpoint: http://192.168.226.100:9411/api/v2/spans

针对项目的改造增加的代码、配置基本上就完成了

通过调用服务验证ZipKin

通过如下图可以看到每个链路总消耗时间，

查看单个链路在每个服务上锁消耗的时间，然后可以再去排查对应的问题所在

总结

​ 通过Zipkin跟踪整条链路消耗时间情况来快速定位问题，尤其是整个项目是在微服务化的情况下运行。那么这种工具就是必备的了。自己也可以针对Zipkin下载源码来打包，可以二次开发，增加对应的预警功能等。后续上生产后，会把存储改造为物理存储，即Cassandra、和 ElasticSearch这种nosql数据库。

分享到 评论