欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

Flink-on-yarn

程序员文章站 2022-07-14 13:42:22
...

介绍

官网下载

flink集群地址

  • ddcve: master
  • dddve: slave
  • ddeve: slave

解压

tar -xzvf flink-1.6.1-bin-hadoop28-scala_2.11.tgz
mv flink-1.6.1 flink
cd flink

修改配置

  • 指定master
    • vim conf/flink-conf.yaml
    jobmanager.rpc.address: ddcve   # 指定master
  • 指定slaves
echo -e "dddve\nddeve" > conf/slaves

添加yarn配置项

  • hadoop/etc/hadoop/yarn-site.xml
<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>8</value>
</property>

启动flink

./bin/start-cluster.sh
  • 输出结果如下
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host ddcvc.
Starting standalonesession daemon on host ddcve.
Starting taskexecutor daemon on host dddve.
Starting taskexecutor daemon on host ddeve.

flink on yarn

以下操作都在slave

  • 启动yarn-session
./bin/yarn-session.sh -d -n 4 -tm 2048 -s 4
  • 提交job
# 先在hdfs创建输入文件
hadoop fs -copyFromLocal /tmp/passwd /tmp/passwd

# 提交
./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/passwd

flink with zookeeper

  • 修改conf/flink-conf.yaml
high-availability: zookeeper
high-availability.storageDir: hdfs:///flink/ha/
high-availability.zookeeper.quorum: ddcva:2181,dddva:2181,ddeva:2181
recovery.zookeeper.path.root: /flink
high-availability.zookeeper.client.acl: open
state.backend: filesystem
  • 修改conf/masters
ddcvc:8081
ddcve:8081