News

Installing Deephaven without Docker | Deephaven

stp2y0 Comments
Installing Deephaven without Docker | Deephaven


Typically, when you start Deephaven, you would go through these steps, which
use Docker and Docker Compose to set up a Deephaven server and its dependencies.

However, running inside of Docker is not the only way you can set up Deephaven. Today, I’m going to walk through the
steps necessary to get a Deephaven Community Core server running on
a fresh install of Linux.

We have now simplified deployment via a single application and Deephaven can more easily be run without Docker. If you have any questions, reach out on our Community Slack.

Here’s the kernel I just installed on a recently flashed machine:

/$ uname -a
Linux wote 5.11.0-34-generic

Pre-Installed dependencies​

Although we’ll install most dependencies as we go, a few tools we need were already
installed on my machine:

Program Version Installed
git 2.30.2
tar 1.34

Creating a deephaven user​

Deephaven’s server is very powerful. You probably already know, but
users can write and execute custom code within a REPL. You should
assume that any user will have the same privileges as the user that owns
the JVM process. For a little extra protection, I like to create
a new user that has relatively limited access to this system.

Add the JVM flag -Ddeephaven.console.disable=true to disable all REPL access.

/$ sudo adduser deephaven

Let’s create a root level directory and transfer ownership to our new user:

/$ sudo mkdir /deephaven
/$ sudo chown -R deephaven:deephaven /deephaven

Ensure this directory is writable by the deephaven user group and that
new files inherit the group ownership:

/$ sudo chmod g+ws /deephaven

Let’s add our user to the deephaven group.

/$ sudo usermod -aG deephaven $USER

Restart your terminal to allow the group change to take effect.

You can verify that your user is now in the deephaven group:

/$ groups $USER
nate : nate adm cdrom sudo dip plugdev lxd deephaven

The following section can be skipped if you don’t intend to build on this
machine. Instead, you can upload and install artifacts built from another host
if that suits your style better. Nothing in this section is required to run a
Deephaven instance.

If you aren’t building on this machine, jump to the Nginx section.

Installing JDK 8​

Deephaven requires jdk8 to build.

/$ sudo apt install openjdk-8-jdk
[ ... ]

/$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Installing Docker​

Although our end product won’t be using Docker, the build system still utilizes
Docker to ensure that any of our users can build the same artifacts that we do.

Here I follow the official instructions
as listed for the hirsute 21.04 distribution:

  1. Enable apt to use a repository over https:
/$ sudo apt install apt-transport-https 
    ca-certificates                     
    curl                                
    gnupg                               
    lsb-release
  1. Install Docker’s official GPG key:
/$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | 
    sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
  1. Add Docker’s stable hirsute repository:
/$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] 
       https://download.docker.com/linux/ubuntu 
       $(lsb_release -cs) stable" | 
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  1. Install the latest and greatest version of Docker (see instructions for alternatives):
/$ sudo apt update
/$ sudo apt install docker-ce docker-ce-cli containerd.io
  1. Run docker hello-world test (as super user):
/$ sudo docker run hello-world

This is the response I received:

Docker response 
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:61bd3cb6014296e214ff4c6407a5a7e7092dfa8eefdbbec539e133e97f63e09f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image, which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Sudo-less Docker​

The docker group grants privileges equivalent to the root user. Every account
added to the group can elevate privileges to root without a password.

Let’s follow Docker’s post-install instructions
that enable sudo-less use of Docker.

  1. Create the docker group:
  1. Add your user to the docker group:
/$ sudo usermod -aG docker $USER
  1. Restart your terminal to allow the group change to take effect:
/$ groups $USER
nate : nate adm cdrom sudo dip plugdev lxd deephaven docker
  1. Run the docker hello-world test (no longer as a super user!):
/$ docker run hello-world

Building​

Let’s obtain the source!

/$ cd /deephaven


/deephaven$ git clone https://github.com/deephaven/deephaven-core

/deephaven$ cd deephaven-core

Let’s check out the most recently released version of Deephaven (as of this post):

/deephaven$ git checkout v0.4.1

If this isn’t your first trip through this post, then you may want to
clean any artifacts in the target build directory.

/deephaven/deephaven-core$ rm -rf ./server/server/native/build

Finally, we build and package Deephaven Community Core:

/deephaven/deephaven-core$ ./gradlew :server-server-native:build

To serve-up the static assets of our web-client-ui,
Deephaven’s Docker setup uses nginx. We’ll do the same.

You can follow Ubuntu’s nginx tutorial to
learn more about nginx.

Install nginx:

/$ sudo apt install nginx-full

Deephaven’s notebook editor and layout manager use nginx’s webdav module, which is not installed via the nginx package. This is why we install nginx-full here.

Configure Deephaven as an nginx enabled-site. Let’s seed our configuration from the existing
configuration that Deephaven’s docker compose uses:

/$ sudo cp /deephaven/deephaven-core/web/client-ide/nginx/default.conf /etc/nginx/sites-enabled/deephaven

Edit the file to your taste. In particular:

  • I’m changing the port from the default HTTP port of 80 to 1818. This helps us avoid using HTTP when we want to HTTPS.
  • I’m replacing the location root from /usr/share/nginx/html to /deephaven/html.
  • I’m replacing the storage location for notebooks/layouts from /data to /deephaven/storage.
  • I’m replacing the server_name from localhost to a DNS entry on my personal domain that resolves to its IP address on my internal network.

My version of this file can be found here.

We also need to configure the notebooks limit:

  • Place dav_ext_lock_zone zone=notebooks:10m; near the end of the http section of /etc/nginx/nginx.conf.

I remove the default site that is installed with nginx. It binds to port 80, and I would prefer this machine to only
respond to HTTPS requests:

/$ sudo rm /etc/nginx/sites-enabled/default

Let’s create the necessary folders to make the configuration valid:

/$ mkdir /deephaven/html
/$ mkdir /deephaven/storage
/$ mkdir /deephaven/storage/notebooks
/$ mkdir /deephaven/storage/layouts

Let’s also add the www-data user to our deephaven group, so that it also can read and write to /deephaven:

/$ sudo usermod -aG deephaven www-data

Finally, let’s restart the nginx service:

/$ sudo service nginx restart

If you’re successful, then hitting port 80 responds with This site cannot be reached, and hitting port 1818 should give you a 404.

Deephaven uses Envoy to send static content requests to nginx and gRPC
requests to the server process. To install, I’ll follow along with the official directions:

/$ sudo apt update
/$ sudo apt install apt-transport-https gnupg2 curl lsb-release
/$ curl -sL 'https://deb.dl.getenvoy.io/public/gpg.8115BA8E629CC074.key' | sudo gpg --dearmor -o /usr/share/keyrings/getenvoy-keyring.gpg
/$ 
/$ echo a077cb587a1b622e03aa4bf2f3689de14658a9497a9af2c427bba5f4cc3c4723 /usr/share/keyrings/getenvoy-keyring.gpg | sha256sum --check
/$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/getenvoy-keyring.gpg] https://deb.dl.getenvoy.io/public/deb/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/getenvoy.list
/$ sudo apt update
/$ sudo apt install -y getenvoy-envoy

Let’s grab the Envoy configuration that the Docker setup uses as a starting point:

/$ mkdir /deephaven/config
/$ cp /deephaven/deephaven-core/envoy/contents/envoy.yaml /deephaven/config
  1. Remove references to grpc-proxy:

At the time of this writing, the Docker set up has a third utility called grpc-web. For
our setup, we won’t need it. gRPC-web enables gRPC over websockets, but we are going to utilize
HTTP2, which requires SSL.

It is not strictly necessary to remove them from your configuration. I don’t need them, so I prefer to remove them.

  • Remove the route entry that forwards calls to grpc-proxy (from listener_0).
  • Remove the cluster named grpc-proxy.

  1. Replace listener_0‘s socket_address port from 10000 to default HTTPS port of 443.

  2. Configure SSL.

Obtaining an SSL certificate is out of scope for this post. However, there are
numerous tutorials on how to generate a self-signed certificate. For our purposes, I generated a wildcard
cert for my personal domain, at no cost, with the help of certbot.
This specific path required that I add TEXT DNS entries to the domain that I control to prove ownership.

I added this SSL configuration block under listener_0‘s filter_chains (at the same level as filters):

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
  '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
  common_tls_context:
    alpn_protocols: ['h2']
    tls_certificates:
      - certificate_chain:
          filename: /deephaven/config/fullchain.pem
        private_key:
          filename: /deephaven/config/privkey.pem

It’s important to keep tight restrictions on your SSL certificate’s private key; it should not be readable by
the deephaven user.

  1. Change the endpoint.address.socket_address.address for both server and web clusters. I’ve set these
    both to localhost.

  2. Change the socket_address.port_value for the web cluster from 80 to the same port you configured nginx
    to listen to. (I used port 1818.)

  3. Either remove the admin port entry, or restrict access to localhost (or 127.0.0.1).

For your convenience, here are the initial
and final configurations.

Finally, let’s start Envoy:

/$ sudo envoy -c /deephaven/config/envoy.yaml

We should now be able to navigate to the https version of our page and land on that same 404 that nginx served us earlier.

Automating the launch of Envoy is left as an exercise to the reader.

Now that we have built the native distribution, we can populate our host with
the necessary artifacts and resources:

/$ export VERSION=0.4.1
/$ mkdir /deephaven/lib
/$ tar xvf /deephaven/deephaven-core/server/server/native/build/distributions/server-server-native-$VERSION.tar -C /deephaven/lib --strip-components=2 "server-server-native-$VERSION/lib/"

To redeploy, be sure to clear that lib directory before extracting the new jars:

/$ rm -rf /deephaven/lib/*

Let’s initialize our runtime logback.xml. Note that you can start with the template
provided at /deephaven/deephaven-core/server/server/native/build/resources/logback.xml.
If you choose to roll your own, be sure to copy the LogBufferAppender section or else
REPL users won’t get logs. I prefer rolling logs, so I went with this:

/$ mkdir /deephaven/resources
/$ cat > /deephaven/resources/logback.xml
<configuration debug="false">

  <property name="LOG_FILE" value="dhvn" />
  <property name="LOG_DIR" value="/deephaven/logs/" />
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${LOG_DIR}/${LOG_FILE}.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>${LOG_DIR}/archive/%d{yyyyMMdd}/%d{yyyyMMdd-HH-}${LOG_FILE}.gz</fileNamePattern>
      <totalSizeCap>300MB</totalSizeCap>
    </rollingPolicy>

    <encoder>
      <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z', UTC} | %green(%-20.20thread) | %highlight(%5level) | %yellow(%-25.25logger{25}) | %m%n</pattern>
    </encoder>
  </appender>

  <!-- System.out / System.err may be redirected (and captured by LogBuffer).
       By referencing PrintStreamGlobalsConsole, we can be sure that we avoid that redirection so
       we don not double up messages to the LogBuffer. -->
  <appender name="STDOUT" class="io.deephaven.logback.PrintStreamGlobalsConsole">
    <encoder>
      <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z', UTC} | %green(%-20.20thread) | %highlight(%5level) | %yellow(%-25.25logger{25}) | %m%n</pattern>
    </encoder>
  </appender>

  <appender name="LOGBUFFER" class="io.deephaven.logback.LogBufferAppender">
    <encoder>
      <!-- LogBufferRecord has timestamp and level, so no need to encode -->
      <pattern>%-20.20thread | %-25.25logger{25} | %m</pattern>
    </encoder>
  </appender>

  <root level="info">
    <appender-ref ref="FILE" />
    <appender-ref ref="LOGBUFFER" />
  </root>
</configuration>

Here’s a mini-script that wraps starting up the Deephaven server:

#!/usr/bin/env bash

java_home=$(update-java-alternatives -l | grep -w "1.8.0" | sed 's/  */ /g' | cut -f 3 -d ' ')

pushd /
$java_home/bin/java -server                         
         -XX:+UseG1GC                               
         -XX:MaxGCPauseMillis=100                   
         -XX:+UseStringDeduplication                
         -Xmx8G                                     
         -XshowSettings:vm                          
         -cp /deephaven/resources/:/deephaven/lib/* 
         -Ddeephaven.console.type=groovy            
         io.deephaven.grpc_api.runner.Main

We’ll utilize the web-client-ui build that deephaven-core depends on.

  1. Build and install the jsapi:
/$ cd /deephaven/deephaven-core
/deephaven/deephaven-core$ rm -rf /deephaven/html/jsapi
/deephaven/deephaven-core$ ./gradlew webpackSources
/deephaven/deephaven-core$ mkdir /deephaven/html/jsapi
/deephaven/deephaven-core$ cp proto/raw-js-openapi/build/dhapi/dh-internal.js /deephaven/html/jsapi
/deephaven/deephaven-core$ ./gradlew prepareCompose
/deephaven/deephaven-core$ cp -rf /deephaven/deephaven-core/web/client-ide/build/docker/dhapi/* /deephaven/html/jsapi/
  1. Extract Deephaven’s Code Studio from the released npm package:

We’ll use npm to download Deephaven’s code-studio. Alternatively, you can download the tar file directly from
npmjs.

Let’s check the version that deephaven-core is going to expect:

/$ cat /deephaven/deephaven-core/web/client-ui/Dockerfile

The results tell us this was built with version 0.3.1:

FROM docker.io/library/node:14.16.0 AS node
WORKDIR /usr/src/app

FROM node as build
ARG WEB_VERSION=0.3.1


RUN set -eux; 
    npm pack @deephaven/code-studio@${WEB_VERSION}; 
    tar --touch -xf deephaven-code-studio-${WEB_VERSION}.tgz; 
    rm deephaven-code-studio-${WEB_VERSION}.tgz;%

Fetch and unpack the Deephaven IDE:

/$ cd /deephaven/html
/deephaven/html$ rm -rf ide
/deephaven/html$ mkdir ide
/deephaven/html$ npm pack @deephaven/code-studio@0.3.1
/deephaven/html$ tar zxvf deephaven-code-studio-0.3.1.tgz -C ide package/build --strip-components=2
/deephaven/html$ rm deephaven-code-studio-0.3.1.tgz

If you’ve been following along, your server is now running nginx for static assets, envoy to enable SSL and
route between nginx and deephaven, and a JVM running deephaven in Groovy REPL mode. See
below for instructions to install a custom version of the web-client. Integrating with Python is a bit involved
and therefore out of scope for this post.

Further reading​

Build the web client​

Testing out changes to the web-client-ui is quite easy to accomplish in our new
environment. Let me walk you through the steps.

  1. The web client uses npm to build. Let’s install it:
  1. At the time of this article, this build requires npm version 6.14.14. Let’s install that now:

Restart your terminal to allow the npm version change to take effect.

  1. Let’s checkout the source:
/$ cd /deephaven
/deephaven$ git clone https://github.com/deephaven/web-client-ui
  1. Let’s prepare the repository to build the release artifacts:
/deephaven$ cd web-client-ui
/deephaven/web-client-ui$ npm install
  1. Let’s produce the release artifacts:
/deephaven/web-client-ui$ npm run build
  1. Finally, let’s replace the existing ide with the one we just built:
/deephaven/web-client-ui$ rm -rf /deephaven/html/ide
/deephaven/web-client-ui$ cp -rf packages/code-studio/build /deephaven/html/ide
  1. After a quick refresh of the browser, you’re using your own build of the web client!



Source link
lol

By stp2y

Leave a Reply

Your email address will not be published. Required fields are marked *

No widgets found. Go to Widget page and add the widget in Offcanvas Sidebar Widget Area.