搜索

DeBezium


发布时间: 2022-11-24 18:36:04    浏览次数:34 次

DeBezium

DeBezium简介
    DeBezium:功能远远强大与canal、maxwell。构建与kafka之上。
    优点:1.处理大容量的数据  2.监控多种数据库:mysql、MongoDB、PostgreSQL、
        SQL Server。canal、maxwell限于(mysql)
    方式一:
        Source connector:(从关系型数据库获取数据)
        Sink   connector:(将数据输出,一般需要做数据处理分析,所以一般不用)
    方式二:部署成Server(当前只能sink到kinesis)
    方式三:嵌入式引擎(Flink CDC就是内置了DeBezium)
    文档:https://debezium.io/documentation/reference/2.0/

mysql连接器

下载安装 mysql连接器
    mkdir -p  /opt/debezium/connector/
    cd  /opt/debezium/connector/
    wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/1.5.4.Final/debezium-connector-mysql-1.5.4.Final-plugin.tar.gz
    tar -xvf debezium-connector-mysql-1.5.4.Final-plugin.tar.gz
    cp /opt/debezium/connector/debezium-connector-mysql/*.jar /opt/kafka/kafka/libs/
修改kafka配置
    cd /opt/kafka/kafka/config/
    vim connect-distributed.properties
    bootstrap.servers=localhost:9092
    group.id=connect-cluster
    key.converter=org.apache.kafka.connect.json.JsonConverter
    value.converter=org.apache.kafka.connect.json.JsonConverter
    plugin.path=/opt/debezium/connector/
重启kafka
    cd /opt/kafka/kafka/bin/
    ./kafka-server-stop.sh;./zookeeper-server-stop.sh;
    ./zookeeper-server-start.sh -daemon /opt/kafka/kafka/config/zookeeper.properties  
    ./kafka-server-start.sh -daemon /opt/kafka/kafka/config/server.properties
    ###启动 kafka connector
    ./connect-distributed.sh -daemon /opt/kafka/kafka/config/connect-distributed.properties
检测
    ###检测 kafka connector 是否正常
    curl -H "Accept:application/json" localhost:8083/connector-plugins
    curl -H "Accept:application/json" localhost:8083/connectors
    ###注册 kafka connector
    curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{"name":"mysql_connector","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","database.hostname":"localhost","database.port":"3306","database.user":"root","database.password":"123456","database.server.id":"184","database.server.name":"student","test.student":"inventory","database.history.kafka.bootstrap.servers":"localhost:9092","database.history.kafka.topic":"dbhistory.student","include.schema.changes":"true"}}'
    ###删除 kafka connector
    curl -i -X DELETE localhost:8083/connectors/inventory-connector
辅助命令
    ##删除topics
    ./kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic events
    ##创建topics
    ./kafka-topics.sh --create --topic events --bootstrap-server localhost:9092
    ##desc topics
    ./kafka-topics.sh --describe --topic events --bootstrap-server localhost:9092


    ##创建命令行生产者
    ./kafka-console-producer.sh --topic events --bootstrap-server localhost:9092
    ##打印topic
    ./kafka-console-consumer.sh --topic events --from-beginning --bootstrap-server localhost:9092

    ./kafka-console-consumer.sh --topic dbhistory.student --from-beginning --bootstrap-server localhost:9092


    cd /opt/mysql/mysql/bin
    ./mysql -u root -p'123456'
    use test;
免责声明 DeBezium,资源类别:文本, 浏览次数:34 次, 文件大小:-- , 由本站蜘蛛搜索收录2022-11-24 06:36:04。此页面由程序自动采集,只作交流和学习使用,本站不储存任何资源文件,如有侵权内容请联系我们举报删除, 感谢您对本站的支持。 原文链接:https://www.cnblogs.com/wuxiaolong4/p/16806563.html