Skip to main content

7. (Optional) Check out Spark streaming

If you have Spark streaming enabled in your template, you can now test it out.

7.1 Check the streaming specification​

In you IDE please open the file streaming.yaml in the root of your directory. The file will look something like this:

streamingApplications:
- name: console
  sparkSpec:
    numberOfExecutors: 1
    driverInstanceType: mx.micro
    executorInstanceType: mx.micro
    application: "local:///opt/spark/work-dir/src/$PROJECT_NAME/StreamingApp.scala"
    applicationArgs:
      - --env
      - "{{ .Env }}"

As you can see a streaming application called console is defined. It will run the code in the StreamingApp.scala file.

The configuration options of the streaming.yaml can be found in the docs on Spark Streaming.

7.2 Streaming UI​

Since we deployed the project before our spark streaming application is actually already running!

To see it running you should open the Conveyor UI, navigate to your environment and press the streaming application tab.

You should see a streaming application for your project running. Clicking on the application brings you to the logs, you can also open the spark UI.

Streaming Applications in the Conveyor UI