MapR Streams Replication

MapR Streams Replication Setup

hadoop fs -mkdir /mapr/SourceCluster/maprstream/

hadoop fs -mkdir /mapr/SourceCluster/maprstream/customer/

hadoop fs -mkdir /mapr/DestinationCluster/maprstream/

hadoop fs -mkdir /mapr/DestinationCluster/maprstream/customer/

Step 1) Create source stream

maprcli stream create -path /mapr/SourceCluster/maprstream/customer/src -produceperm p -consumeperm p -topicperm p -defaultpartitions 3

Step 2) Create replica Stream

maprcli stream create -path /mapr/DestinationCluster/maprstream/customer/dst -produceperm p -consumeperm p -topicperm p -defaultpartitions 3

Step 3) Copy meta from src to replica stream

maprcli stream create -path /mapr/DestinationCluster/maprstream/customer/dst -copymetafrom /mapr/SourceCluster/maprstream/customer/src

Step 4) Register the replica as a replica of the source stream by running the maprcli stream replica add command

maprcli stream replica add -path /mapr/SourceCluster/maprstream/customer/src -replica /mapr/DestinationCluster/maprstream/customer/dst -paused true

Step 4a) Verify that you specified the correct replica by running the maprcli stream replica list command

maprcli stream replica list -path /mapr/SourceCluster/maprstream/customer/src -json

Step 5) Authorize replication between the streams by defining the source stream as the upstream stream for the replica

maprcli stream upstream add -path /mapr/DestinationCluster/maprstream/customer/dst -upstream /mapr/SourceCluster/maprstream/customer/src

Step 5a) Verify that you specified the correct source stream by running the maprcli stream upstream list command 

maprcli stream upstream list -path /mapr/DestinationCluster/maprstream/customer/dst -json

Step 6) Load the replica with data from the source stream by using the mapr copystream utility.

mapr copystream -src /mapr/SourceCluster/maprstream/customer/src -dst /mapr/DestinationCluster/maprstream/customer/dst

Command to insert data to source stream:

echo "Hello, World" | /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic /mapr/SourceCluster/maprstream/customer/src:test

Command to count number of messages on Stream

Source : mapr streamanalyzer -path /mapr/DestinationCluster/maprstream/customer/src

Destination : mapr streamanalyzer -path /mapr/DestinationCluster/maprstream/customer/dst

Command to insert data to source stream:

echo "Hello, World" | /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic /mapr/SourceCluster/maprstream/customer/src:test

Command to count number of messages on Stream

Source : mapr streamanalyzer -path /mapr/DestinationCluster/maprstream/customer/src

Destination : mapr streamanalyzer -path /mapr/DestinationCluster/maprstream/customer/dst

Command to resume and stop replication

maprcli stream replica resume -path /mapr/SourceCluster/maprstream/customer/src -replica /mapr/DestinationCluster/maprstream/customer/dst

maprcli stream replica pause -path /mapr/SourceCluster/maprstream/customer/src -replica /mapr/DestinationCluster/maprstream/customer/dst

Gateway List

maprcli cluster gateway list

cluster  gatewayConfig

sn1      sn1

Env :

mapr-kafka-0.9.0.201611301215-1.noarch

mapr-core-5.2.0.39122.GA-1.x86_64

NOTE - Following are the documentation for more details

Gateway - http://maprdocs.mapr.com/51/#Gateways/ConfiguringMapRGatewaysForTRAndI.html

Streams Replication - http://maprdocs.mapr.com/51/#MapR_Streams/setting_up_ms_repl.html