ALU

Linux

Comparing Windows and Linux SQL Containers

By several measures, Windows SQL Server containers offer better enterprise support than Linux MySQL or Postgres containers. SQL Server containers provide more backward compatibility, and support for existing apps, storage arrays, and infrastructure.

Windocks has evolved as an independent port of Docker’s open source project to include database cloning, a web UI, secrets store, and other capabilities. These capabilities are customer driven, and seem to diverge from Linux mainstream development. This article takes looks at the capabilities being driven by Windows customers. Full disclosure, I am a principal of Windocks, and this article focuses on the Windows-based SQL Server containers provided by Windocks.

Original Link

Running SQL Server on a Linux Container Using Docker for Windows

Recently, I have been investigating what all the fuss is about Docker and it has been well worth my time as Docker is pretty awesome for automating stuff.

My development environment has typically required installing SQL Server. SQL is a bit of a beast with lots of options and takes time to set up how you want.

Original Link

What Are the Prerequisites to Learn Cloud Computing AWS?

What is Amazon Web Services (AWS)?

Amazon Web Services, commonly called AWS, is an extensive and secure cloud services platform presented by Amazon. The AWS Cloud or Amazon cloud provides a wide range of facilities services, such as storage options, processing power, networking and databases to businesses, helping them scale and expand. Amazon gives its services on-demand with pay-as-you-go pricing plan.

AWS promotions were first launched in 2006 and presently it is the leading cloud services supplier.

Original Link

Containers Are and Will Be the New Linux

Linux is the operating system which has revolutionized data centers over last 2 decades, and today it is the undisputed leader in application hosting platforms. It’s very hard to imagine deploying any mission critical production workloads to any other platform than Linux.

A similar revolution in packaging, deploying, and hosting applications was started a few years ago, when Docker made the Linux containers popular. After that, the growth in container adoption across the industry was exponential and it’s multiplying with each passing day.

Original Link

Managing Your Alibaba Cloud Linux Server From the CLI

Alibaba Cloud offers flexible Elastic Compute Service (ECS) products for running powerful Virtual Private Servers (VPS). With their simple and transparent pricing starting from $4.50/month, you can set up a remote server to cater for all your cloud hosting needs.

You can deploy an Alibaba Cloud ECS instance in a few minutes right from their friendly console and scale resources up or down depending on your cloud computing needs.

Original Link

Use Docker Instead of Kubernetes

Today we are all talking about containers and container-based infrastructure.  But what is this container technology? And how does it solve today problems?

I am using containers myself and of course, I am fascinated by this server technology. Containers can really simplify things. After more than 20 years in building server applications, I have experienced many problems very closely.

Original Link

How to Use the AWS CLI

The AWS Command Line Interface (CLI) is for managing your AWS services from a terminal session on your own client, allowing you to control and configure multiple AWS services.

So you’ve been using AWS for awhile and finally feel comfortable clicking your way through all the services. However, you may have noticed that there is more to AWS than the default eye-catching browser console.

Original Link

Corwin on Containers [Video]

Corwin Brown (BloomReach) presents a deep dive into the history of containers as well as an introduction to how they work under the covers. This includes a discussion around Control Groups and Process Namespaces, as well as touching on some underlying syscalls, such as Fork and Clone. 

Original Link

Customization and Externalization of Property Files in Spring Boot

By now, you must know why we use Spring boot. Yep, that’s it — because it’s developer friendly! As the features and complexity of the Spring framework are growing, the configuration of your project becomes a more tedious task. To counter the issues developers on the Pivotal team come up with, the Spring Boot project and Spring Jira are improving containerless applications, which triggered the idea of Spring Boot.

Let’s have a look at the configuration for the customization and externalization of property files that can be done in a Spring Boot application.

Original Link

How to Set Up Your First CentOS 7 Server on Alibaba Cloud

Alibaba Cloud Elastic Compute Service (ECS) provides a faster and more powerful way to run your cloud applications as compared to traditional physical servers. With ECS, you can achieve more with the latest generation of CPUs as well as protect your instance from DDoS and Trojan attacks.

In this guide, we will talk about the best practices for provisioning your CentOS 7 server hosted on an Alibaba Cloud Elastic Compute Service (ECS) instance.

Original Link

Install Arch Linux on Windows 10 Hyper-V

A few months back, I upgraded my laptop to a new Microsoft Surface Book 2 that has all these nice features, like a detachable screen. My previous laptop and current desktop run Arch Linux as the main OS. My work is mostly done in the IDE Visual Studio Code or shell. VCS runs well on Windows 10, but shell, I don’t think I want to invest too much time in PowerShell and WSL is not exactly the best working environment for me.

I decided to deploy Arch Linux on a Windows Hyper-V virtual machine for my bash needs as I am not yet sure if I want to completely replace Windows as the main desktop on my Surface Book 2, since I still like the detachable screen and Windows 10 face authentication and the VR kit. Maybe, in the near future, I will deploy Arch Linux beside Windows 10.

Original Link

Docker as Package Manager for Linux

I first tried Linux in 1995 and it was Red Hat Linux distribution version 2.1. It was quite a challenge to set it up to work with Intel 80486DX4. In the 90’s, installing something into Linux took some time to get used to after Windows “anybody *.tar.gz + dependencies."

Red Hat was one of the first to come with integrated package management system RPM that was suppose to solve dependencies. Once you start using it, more and more of the rpm installation’s limitations start to creep in, a phenomenon called Dependency hell. Debian was a good alternative with DEB packages.

Original Link

Repair SQL Server on Linux After an Ubuntu Distribution Upgrade [Snippet]

SQL Server 2017 is supported on Ubuntu 16.04 LTS (Long-Term Support), however that version of Ubuntu Linux is now more than two years old, so you may be tempted to update Ubuntu to a more modern release such as Ubuntu 18.04 LTS (Bionic Beaver). Unfortunately (as of this writing), SQL Server 2017 is not supported on later versions of Ubuntu.

(Note: Ubuntu releases new distributions every six months. The version number 16.04 means that it was released in April 2016.)

The more adventurous among us who use SQL Server 2017 in a development environment — where we need to keep the Ubuntu distribution as up to date as possible — might run into the following issue:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help resolve the situation: The following packages have unmet dependencies:
mssql-server : Depends: openssl (<= 1.1.0)
E: Unable to correct problems, you have held broken packages.

I would like to share this very handy answer written by langioletto that I found on Ask Ubuntu, which resolves this dependency issue.

WARNING: Downgrading packages is dangerous, especially in libraries like OpenSSL.  Do not run this in a production environment!

apt-get install -y libjemalloc1 libsss-nss-idmap0 libc++1 gawk curl
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
add-apt-repository "$(curl https://packages.microsoft.com/config/ubuntu/16.04/mssql-server-2017.list)"
add-apt-repository "$(curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list)"
wget http://archive.ubuntu.com/ubuntu/pool/main/c/ca-certificates/ca-certificates_20160104ubuntu1_all.deb
dpkg -i ca-certificates_20160104ubuntu1_all.deb
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.0.2g-1ubuntu4_amd64.deb
dpkg -i openssl_1.0.2g-1ubuntu4_amd64.deb
apt install -y libcurl3
apt-get update
apt-get install -y mssql-server

Once SQL Server is installed, you will have to run sudo /opt/mssql/bin/mssql-conf setup to make sure SQL Server is configured correctly. Fortunately, SQL Server 2017 on Linux is non-destructive with any databases you already had from previous installations, so the service should start up happily by itself and all your databases should be usable.

Thanks, langioletto!

Share your Linux development environment tips and tricks with me on Twitter at @bornsql.

Original Link

How to Make It Easy and Simple to Start Java Processes in Linux/Docker

One of our colleagues works as the DevOps-engineer, and he often deals with the automation of the installation and configuration of a variety of IT-systems in various environments, from containers to the cloud. He used to work with many systems based on the Java-stack: from small (like Tomcat) to large (Hadoop, Cassandra, etc.).

Almost every such system, even the simplest, for some reason had a complex unique launch system. At a minimum, these were multi-line shell scripts, as in Tomcat, and even entire frameworks, as in Hadoop. Our current “patient” in this series, inspired us to write this article — the repository of Nexus OSS 3 artifacts, the launch script which takes ~ 400 lines of code.

Opacity, redundancy, entanglement of startup scripts creates problems even when manually installing one component on the local system. And now imagine that the set of such components and services needs to be packaged in a Docker container, simultaneously writing another layer of abstraction for more or less adequate orchestration, deployed in the Kubernetes cluster and implemented this process in the form of CI / CD-payline.

In short, let’s look at the example of the Nexus 3 mentioned above, how to return from the labyrinth of shell scripts to something more similar to java -jar <program.jar>, given the convenient modern DevOps-tools.

Why Such Complexity?

In a nutshell, in ancient times, when UNIX was not interrogated at the mention of “in the sense of Linux?”, there were no Systemd and Docker, etc. To manage the processes, we used portable shell scripts (init scripts) and PID-files. Init scripts set the necessary environment settings, which in different UNIX-s were their own, and, depending on the arguments, they started the process or restarted/stopped it using the ID from the PID-file. The approach is simple and straightforward, but these scripts stop working with every non-standard situation, requiring manual measurement, do not allow running several copies of the process…but not the point.

So, if you look closely at the above-mentioned startup-scripts in Java-projects, you can see in the obvious signs of this prehistoric approach, including even mentioning SunOS, HP-UX and other UNIX-s. Typically, these scripts do something like:

  • Use the POSIX shell syntax with all its crutches for UNIX/Linux portability
  • Determine the version and release of the OS via uname, /etc/* release, etc.
  • Look for JRE/JDK in the secluded corners of the file system and choose the most appropriate version by tricky rules, sometimes even specific for each OS
  • Calculate the numerical parameters of the JVM, for example, the memory size (-Xms, -Xmx), the number of GC streams, etc.
  • Optimize the JVM via -XX-parameters taking into account the specificity of the selected version of the JRE/JDK
  • Finds their components, libraries, paths to them by surrounding directories, configuration files, etc.
  • Configure the environment: ulimits, environment variables, etc.
  • Generate a CLASSPATH loop of type: for f in $ path/*. jar; do CLASSPATH = “$ {CLASSPATH}: $ f”; made
  • The parsing arguments to the command line: start|stop|restart|reload|status|…
  • Collect the Java-command, as a result, it is necessary to execute, from listed above
  • And, finally, execute this Java command. Often, all the same, notorious PID files are used, either explicitly or implicitly, and, nohup, special TCP ports and other tricks from the last century (see Karaf example)

The Nexus 3 startup script is a good example of such a script.

In fact, all the above-scripted logic, as it were, attempts to replace the system administrator, who would install and configure everything manually for a particular system from beginning to end. But in general, any requirements of the most diverse systems are, in principle, impossible to take into account. Therefore, it turns out, on the contrary, a headache, as for developers, you need to support these scripts, and for system engineers, who later need to understand these scripts. From my point of view, it is much easier for a system engineer to understand the JVM parameters once and adjust it as necessary, than every time you install a new system, you must understand the intricacies of its startup scripts.

What To Do?

U – forgive! KISS and YAGNI to us. Especially since there is a 2018 year in the yard, which means that:

  • With very few exceptions, UNIX == Linux
  • The task of process management is solved both for a separate server (Systemd, Docker), and for clusters (Kubernetes, etc.)
  • There was a bunch of convenient configuration management tools (Ansible, etc.)
  • In the administration came and fully entrenched the total automation: instead of manually configuring the fragile, unique “snowflake servers“, you can now automatically create unified reproduced virtual machines and containers with a number of convenient tools, including the Ansible and Docker, mentioned above
  • Tools for collecting runtime statistics are used everywhere, both for the JVM itself (example) and for the Java application (example)
  • And, most importantly, there were specialists: system and DevOps-engineers who know how to use the above technologies and understand how to properly install JVM on a specific system and then adjust it based on the collected runtime statistics

So let’s again go through the functionality of startup-scripts again taking into account the listed items, without trying at the same time to do the work for the system engineer, and remove from there all the “extra.”

  • POSIX shell syntax ⇒ /bin/bash
  • OS version definition ⇒ UNIX == Linux, if there are OS-specific parameters, you can describe them in the documentation
  • Search JRE/JDK ⇒ we have the only version, and this is OpenJDK (well, or Oracle JDK, if it’s really necessary), Java and the company is in the standard system path
  • Calculation of numerical parameters JVM, tuning JVM ⇒ this can be described in the documentation for application scaling
  • Search for your components and libraries ⇒ describe the structure of the application and how to configure it in the documentation
  • Setting the environment ⇒ describe the requirements and features in the documentation
  • Generation of CLASSPATH ⇒ -cp path /to/my/jars/* or even, generally, Uber-JAR
  • Parsing the arguments of the command line ⇒ arguments will not be. Everything except the startup will be taken care of by the process manager
  • Building a Java command
  • Executing a Java command

In the end, we just need to compile and execute a Java command of the form java <opts> -jar <program.jar> using the selected process manager (Systemd, Docker, etc.). All parameters and options (<opts>) are left to the discretion of the system engineer, who will tailor them to a specific environment. If the list of options <opts> is quite long, you can return to the idea of the startup script, but, in this case, as compact and declarative as possible, i.e. which does not contain any program logic.

Example

As an example, let’s see how you can simplify the Nexus 3 launch script.

The easiest option is to not get into the jungle of this script — just run it in real conditions (./nexus start) and look at the result. For example, you can find the full list of arguments for the running application in the process table (via ps -ef), or run the script in debug mode (bash -x ./nexus start) to observe the entire execution process and at the very end a start command.

We finally got the next Java command:

-/usr/java/jdk1.8.0_171-amd64/bin/java -server -Dinstall4j.jvmDir=/usr/java/jdk1.8.0_171-amd64 -Dexe4j.moduleName=/home/nexus/nexus-3.12.1-01/bin/nexus -XX:+UnlockDiagnosticVMOptions -Dinstall4j.launcherId=245 -Dinstall4j.swt=false -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Xms1200M -Xmx1200M -XX:MaxDirectMemorySize=2G -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -XX:+LogVMOutput -XX:LogFile=../sonatype-work/nexus3/log/jvm.log -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Dkaraf.home=. -Dkaraf.base=. -Dkaraf.etc=etc/karaf -Djava.util.logging.config.file=etc/karaf/java.util.logging.properties -Dkaraf.data=../sonatype-work/nexus3 -Djava.io.tmpdir=../sonatype-work/nexus3/tmp -Dkaraf.startLocalConsole=false -Di4j.vpt=true -classpath /home/nexus/nexus-3.12.1-01/.install4j/i4jruntime.jar:/home/nexus/nexus-3.12.1-01/lib/boot/nexus-main.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.apache.karaf.main-4.0.9.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.osgi.core-6.0.0.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.apache.karaf.diagnostic.boot-4.0.9.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.apache.karaf.jaas.boot-4.0.9.jar com.install4j.runtime.launcher.UnixLauncher start 9d17dc87 '' '' org.sonatype.nexus.karaf.NexusMain

First, we apply to it a couple of simple tricks:

We will change /the/long/and/winding/road/to/my/java to java, because it is in the system path
put the list of Java parameters in a separate array, sort it and remove duplicates

We get something more digestible:

JAVA_OPTS = ( '-server' '-Dexe4j.moduleName=/home/nexus/nexus-3.12.1-01/bin/nexus' '-Di4j.vpt=true' '-Di4jv=0' '-Dinstall4j.jvmDir=/usr/java/jdk1.8.0_171-amd64' '-Dinstall4j.launcherId=245' '-Dinstall4j.swt=false' '-Djava.io.tmpdir=../sonatype-work/nexus3/tmp' '-Djava.net.preferIPv4Stack=true' '-Djava.util.logging.config.file=etc/karaf/java.util.logging.properties' '-Dkaraf.base=.' '-Dkaraf.data=../sonatype-work/nexus3' '-Dkaraf.etc=etc/karaf' '-Dkaraf.home=.' '-Dkaraf.startLocalConsole=false' '-XX:+LogVMOutput' '-XX:+UnlockDiagnosticVMOptions' '-XX:+UnlockDiagnosticVMOptions' '-XX:+UnsyncloadClass' '-XX:-OmitStackTraceInFastThrow' '-XX:LogFile=../sonatype-work/nexus3/log/jvm.log' '-XX:MaxDirectMemorySize=2G' '-Xms1200M' '-Xmx1200M' '-classpath /home/nexus/nexus-3.12.1-01/.install4j/i4jruntime.jar:/home/nexus/nexus-3.12.1-01/lib/boot/nexus-main.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.apache.karaf.main-4.0.9.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.osgi.core-6.0.0.jar:/home/nexus/nexus-3.12.1-01/lib/boot/org.apache.karaf.diagnostic.boot-4.0.9.jar:/home/nexus/nexus-3.12.1-01/lib/boot/'
) java ${JAVA_OPTS[*]} com.install4j.runtime.launcher.UnixLauncher start 9d17dc87 '' '' org.sonatype.nexus.karaf.NexusMain

Now you can go deep.

Install4j is such a graphical Java installer. It seems that it is used for the initial installation of the system. We do not need it on the server, we remove it.

We agree on the location of the components and the Nexus data on the file system:

Put the application itself in /opt/nexus- <version>
For convenience, create a symbolic link/opt/nexus ->/opt/nexus- <version>
The script itself will be placed instead of the original as opt/nexus/bin/nexus
all the data of our Nexus will be on a separate file system, mounted as /data/nexus

The very creation of directories and links is the destiny of configuration management systems (for everything about all 5-10 lines in Ansible), so let’s leave this task to system engineers.

Let our script change the working directory to /opt/nexus at startup — then we can change the paths to Nexus components to relative ones.

Options like -Dkaraf. * are the settings for Apache Karaf, the OSGi container, to which our Nexus is obviously packed. Let’s change karaf.home, karaf.base, karaf.etc, and karaf.data according to the placement of components, if possible using relative paths.

Seeing that CLASSPATH consists of a list of jar files that lie in the same lib/directory, we replace the entire list with lib / * (we’ll also have to turn off the wildcard expansion using  set -o noglob ).

Let’s change java  to exec java   so that our script will run Java as a child process (the process manager will not see this child process), but “replace” itself with java (exec description).

Let’s see what happened:

#!/bin/bash JAVA_OPTS=( '-Xms1200M' '-Xmx1200M' '-XX:+UnlockDiagnosticVMOptions' '-XX:+LogVMOutput' '-XX:+UnsyncloadClass' '-XX:LogFile=/data/nexus/log/jvm.log' '-XX:MaxDirectMemorySize=2G' '-XX:-OmitStackTraceInFastThrow' '-Djava.io.tmpdir=/data/nexus/tmp' '-Djava.net.preferIPv4Stack=true' '-Djava.util.logging.config.file=etc/karaf/java.util.logging.properties' '-Dkaraf.home=.' '-Dkaraf.base=.' '-Dkaraf.etc=etc/karaf' '-Dkaraf.data=/data/nexus/data' '-Dkaraf.startLocalConsole=false' '-server' '-cp lib/boot/*'
) set -o noglob cd /opt/nexus \ && exec java ${JAVA_OPTS[*]} org.sonatype.nexus.karaf.NexusMain

Total of 27 lines instead of 400, and it’s transparent, clear, declarative, with no superfluous logic. If necessary, this script can easily be turned into a template for Ansible/Puppet/Chef and add only the logic that is needed for a particular situation.

This script can be used as ENTRYPOINT in Dockerfile or called in the unit-file Systemd, at the same time having adjusted there ulimits and other system parameters, for example:

[Unit]
Description=Nexus
After=network.target [Service]
Type=simple
LimitNOFILE=1048576
ExecStart=/opt/nexus/bin/nexus
User=nexus
Restart=on-abort [Install]
WantedBy=multi-user.target

Conclusion

What conclusions can be drawn from this article? In principle, it all boils down to a couple of points:

  • Each system has its own purpose, that is, it is not necessary to hammer nails with a microscope.

  • Simplicity (KISS, YAGNI) taxis — to realize only what is needed for this particular situation.

  • And most importantly: it’s cool that there are IT specialists of different profiles. Let’s interact and make our IT systems easier, clearer and better!

Original Link

Database Cloning for Docker SQL 2017 Linux Containers

Windocks has evolved over the past two years from an independent port of Docker’s open-source project to Windows, to providing a complete SQL Server data delivery solution. Windocks supports the creation of cloned data environments based on SQL Server backups or storage arrays, with delivery to Microsoft’s SQL Server containers, instances, and Kubernetes clusters. In this article, we look at how database clones are delivered to SQL Server 2017 on Linux containers.

SQL Server 2017 in Linux containers is drawing attention for the speed and agility of containers, as well as database performance. The approach outlined for here should be particularly useful for upgrade testing, as well as general dev/test use. Clones enable delivery of Terabyte databases in seconds, with full read/write operation, while consuming minimal network and storage.   

Building SQL Server Images With Dockerfiles

Dockerfiles are plain text configuration files that define the SQL Server container, and in Windocks 3.0, can be applied at runtime. Each image can include scores of databases with scripts applied for user/group permissions, data masking, and other needs. 

Windocks 3.0 supports external storage arrays and Windows file system database cloning. Windows-based SQL Server images are built with Full or Differential backups, or database files, with each being a full byte copy of the data. Once created, an image supports creation and delivery of clones in seconds with full read/write support, with each requiring less than 40 MB on delivery.

This Dockerfile defines an image that delivers cloned databases to a SQL Server 2017 Linux container.

Image title

ENVUSE_DOCKERFILE_TO_CREATE_CONTAINER ensures the Dockerfile is applied at runtime for each container/environment. Delivery to SQL Server 2017 Linux containers is accomplished with RUN TargetAttach_MSContainerSqlLinux with the parameters shown. SQL Server clones are built with SETUPCLONINGFULL, DIFF (differential) backups, or RAW database files.  

Support for the delivery of database clones over the network is based on SMB, with a file share created on the Windocks host mapped to the Linux host (c:\windocks\data to /windocks/dataexternal as shown above). The Linux setup involves installing SAMBA, and the Docker daemon configured to allow for remote commands.  

A one-time build creates an image that supports an unlimited number of clones. The example below shows the build, followed by a command, to deliver the clone to a new SQL 2017 container. Most of the parameters involved, including the host IP address and target SQL Server image, are included in the image. Only two parameters are required for container creation, including the target port and SQL SA password. 

Image titleManagement of the combined environment is handled by the Windocks container. When it’s time to refresh the environment, the removal of the Windocks container removes the Linux container and associated mounts.    

Working With Sensitive Credentials

Windocks 3.0 introduces encrypted credential support for Windocks images and containers. The workflow described above involves clear text SQL SA passwords, which is the current practice in the use of SQL Server 2017 on Linux. When working with the Windocks SQL Server containers, credentials can be secured using the following methods:   

  • Windocks containers support Windows authentication.
  • Windocks Windows SQL Server containers are created by cloning a SQL Server instance that is configured for use by the Windocks service. Each container inherits SQL logins configured on the parent instance, enabling users with these accounts.
  • Windocks also includes configurable SQL SA credentials for each created SQL container, including an option for no SA passwords, encrypted SA passwords, or passwords in clear text. The three options are configured in the Windocks config folder, node file. SHOW_SA_PASSWORD=”0” or 1, or 2, for no password, encrypted, or clear text, respectively. Restart the Windocks Service following changes to the Windocks configuration.

Windocks encryption is based on the Windows Data Protection API (DPAPI). To encrypt a password, navigate to \Windocks\bin and open a command prompt and enter “encrypt.” The program prompts for a credential string and writes the hashed result to encrypted.txt in the same directory. Open the text file and copy the string into the Dockerfile, in this example we reference the password for an ArrayPassword parameter:

ArrayPassword|1,0,0,0,208,140,157,223,1,21,209,17,140,122,0,192,79,194,1,0, . . .

By incorporating encrypted passwords, either directly in the Dockerfile, the Dockerfiles can be saved and used securely. Once a credential is encrypted the hashed result or environment variable needs to be used in any references to that credential. 

When configured to deliver encrypted credentials, Windocks SQL container SA passwords are delivered in standard Docker client return strings (image below). To unencrypt the credential copy the complete string and save as an encrypted.txt file. RDP to the Windocks server, and copy the encrypted.txt into the \windocks\bin directory. Open a command prompt and enter “decrypt.” 

Image title

The program decrypts the text file and presents the password:

Image title

Working With a Subset of Databases

Users can work with a subset of the databases from an image by using a runtime environment variable: SQL_DB_NAME_OVERRIDES=”dbname1, dbname2”

>docker create -e SQL_DB_NAME_OVERRIDES=”dbname1, dbname2” <imagename>

Working With a Web UI

The Windocks web UI simplifies use for developers and other users. Open a Chrome or Firefox browser and point to the IP address of the Windocks server (local:  127.0.0.1). Images are displayed with required parameters, including the option to work with a subset of desired databases. The image targeting Linux SQL containers only requires user input on the target port and SQL SA password and includes a drop-down selector for working with a subset of the databases in the image.

Image title

Database Cloning for SQL Server Containers and Instances

SQL Server 2017 Linux containers are drawing a lot of attention in a world that is increasingly embracing Linux and open-source technologies. Regardless of the form of SQL Server you use, database cloning is key to enabling an efficient workflow for development and test. Windocks database cloning enables efficient upgrade testing and work with large and complex data environments on the new SQL Server 2017 Linux containers.

You can start exploring these capabilities today by downloading the free Windocks Community Edition http://www.windocks.com/community-docker-windows

Original Link

How Mainframes Save OpEx Through Real Estate, Humans, and Energy Costs

In recent years, mainframes have been viewed as most useful for heavy-lifting within organizations like the Census Bureau or analytics companies which require a lot of data storage, monitoring and retrieval. But times are changing as more enterprise business leaders recognize that having a mainframe instead of a network of smaller machines offers some unique cost reduction benefits over other industry-leading solutions. In fact, mainframes have the potential to reduce OpEx in the following areas: real estate, human capital and energy.

Given the cost reduction that mainframes can offer to businesses today, demand for this monolith of technology is on the rise. Keep reading to find out whether investing in a mainframe is right for your enterprise business requirements.

Traditional Usage and Benefits of a Mainframe

Mainframes are second to none when it comes to processing power. No network of virtualized machines comes close. Today, mainframes can handle a high number of I/O processes and hold exponentially more data than a single PC. This makes them especially useful for large enterprise businesses that require complex computing processes.

Benefits of a Mainframe

In addition to I/O processing and data storage, using mainframes versus other technology has a number of serious benefits to enterprise businesses. Mainframes are known for the following features:

Very Secure

Since most traditional mainframe consumers require that sensitive data is protected, mainframes are built with security in mind. This includes both operating system security and secure hardware.

Wholly Customizable

In most cases, mainframes are wholly customizable to meet the processing needs and work demands of the individual users.

Reliable

To say that supercomputers are reliable would be an understatement. In fact, some are designed to run continuously for as long as 40 years without failure.

Highly Available

Mainframes can be designed to have 99.999% availability.

Easily Scalable

By design, mainframes scale with relative ease. They are designed for plug-and-play system hardware upgrades. Furthermore, new parts of the mainframe can be added and adjusted without interrupting system usage.

Quite Redundant

One of the most important features of mainframes is the potential for redundancy. This is vital for those seeking data housing. And it makes for a smooth disaster recovery process for any enterprise business.

How Are Mainframes Used in Enterprises?

Because of the above benefits, mainframes have traditionally been used for storing critical applications and bulk processing of large amounts of important data.

As opposed to networks of computers that can run on a number of operating systems, traditional mainframes run on a single operating system and operations generally stem from a central processing unit within it that powers all the processes attached to it.

In fact, this is exactly how the machine got its name.

However, modern mainframes are no longer characterized by centralization in the same way traditional machines once were. They are instead designed with redundancy, backwards compatibility and I/O efficiency in mind.

These are the driving characteristics behind modern mainframe design. And because of these characteristics, mainframes evolved into a system where technicians can seamlessly add or swap components without interrupting day-to day-functionality.

The result is business agility and hopefully cost savings.

But it’s not just these traditional qualities of mainframe usage that makes them a good candidate for enterprise servers, storage and processing as you will learn below.

Mainframes and the Introduction of Linux

A monumental breakthrough in mainframe technology came back in 1999 when Linux developed a mainframe compatible operating system.

Linux offered a departure from traditional z/ operating systems. The benefit of Linux is that it’s highly flexible for a number of workloads. You get the full advantage of using Linux OS offerings, including:

  • Simultaneous access to multiple functions of the hardware; and
  • The potential for cloud computing

This makes it a flexible and scalable solution for enterprise businesses.

Unique Cost Reduction Benefits

At the beginning of this article, we mentioned that a few unique benefits really stand out when it comes to making the switch to mainframes with regards to cost reduction. With Linux powering enterprise mainframes, using them can save businesses money in three exceptional ways:

Real Estate Cost Reduction

When you run operations on a network of computers virtualized with software, this scalable model starts to take up a considerable amount of physical space.

This can create an urgent need for a bigger office space that can accommodate a server room. Or it may require expansion to a data center.

However, a signature feature of mainframes is that they scale up instead of out. Indeed, the truth is that one mainframe the size of a commercial refrigerator can accommodate the processing capabilities of thousands of computers. This could mean eliminating entire rooms of hardware in favor of one powerful piece of technology.

Considering that real estate expenses for large businesses tend to be high to begin with, downsizing your infrastructure to a mainframe can offer significant real estate cost savings.

Human Capital Cost Reduction

When you migrate dozens of servers to a mainframe, that’s when you really begin to notice the financial benefit has to offer.

It takes only a small internal IT team of a few people to manage a mainframe and the servers on it. That means you can save money that would otherwise be invested in additional labor to network, secure, monitor and maintain several more machines.

Furthermore, when you start collapsing thousands of servers into one mainframe, the cost savings on labor can be enormous.

Considering salaries for IT managers cost businesses around $135,000 a year, reducing the number of people on an IT workforce can offer a way for businesses to greatly reduce operating expense.

Energy Cost Reduction

Simplifying your system’s infrastructure inevitably leads to cost savings. Whether you are consolidating 45 machines or 4,500, one major benefit is reducing your carbon footprint and saving your business money in the process.

It’s more energy efficient to run your processes out of one mainframe than dozens of computers. But this is not the only way to reduce energy costs with a switch to mainframes. You also save energy when you reduce the amount of real estate required to house dozens of smaller machines. In some cases, that’s a whole building that you no longer need to pay utilities for.

If you’ve been looking for ways to save money and go green, a mainframe can help with this process.

Summary: Reduce OpEx With Mainframes

An old piece of tech is making a comeback in a major way. Once thought as being close to obsolete, mainframes are here to stay. They are scalable, reliable, available and offer competitive cost savings across peripheral budget categories that often go overlooked when we talk about reducing technology costs.

It’s true mainframes are an expensive piece of equipment. But once owned, they offer businesses agility and flexibility. This includes the ability to host a multi-cloud environment off private servers, with little management.

The overall cost savings from big ticket items like real estate, human capital and energy can make up for the cost of a mainframe if the processing of networked computers can be consolidated. And, with Linux capabilities, mainframes are more useful than once thought because they offer increased functionality in a vertical unit, an important benefit for enterprise businesses.

Original Link

Windows Containers in Docker Enterprise Edition 2.0 – Top 7 Questions from the Docker Virtual Event

The recent Docker Virtual Event, Unveiling Docker Enterprise Edition (EE) 2.0, gave us the opportunity to highlight some of the great reasons to adopt a containerization strategy across your entire application portfolio. In Part 1 of this blog, we covered some of the top questions we received about Swarm and Kubernetes orchestration in Docker Enterprise Edition — the world’s leading enterprise-ready container platform. Today, we will cover some of the questions about running Windows containers.

If you missed the live event, don’t worry! You can still catch the recording on-demand here.

Docker Enterprise Edition: Only Fully-Supported Solution for Running Containers on Windows Server 2016

Q: I thought containers were based on Linux processes. How do Windows-based Docker containers work?

A: Docker has been partnering with Microsoft since 2014 to deliver all the same benefits of Docker containers to Windows Server so that customers can easily run .NET and IIS applications in Docker containers. We worked closely together on changes to the Windows Server kernel to support containerization primitives, added Windows Server support to the Docker Engine and CLI and added multi-architecture support for Windows images. The result is Docker containers run natively on Windows Server 2016, leveraging the same CLI commands in Powershell that you use in a Linux-based environment.

Q: Is Windows Server supported with Kubernetes?

A: Support for Windows Server containers in Kubernetes is currently in beta with full support anticipated later this year. To run Windows Server containers in production, customers of Docker Enterprise Edition 2.0 can deploy these workloads with Docker Swarm. This is a feature that has been generally available for over a year.

Q: Can you run full Windows Server clusters?

A: With Docker Enterprise Edition 2.0, you can run clusters of mixed Linux and Windows Server 2016 workers or all Windows Server workers. Note that even if you are running all Windows Server workers, the Docker Enterprise Edition 2.0 management stack is deployed onto Linux-based nodes. For many of our enterprise customers who are Windows shops, they deploy these management nodes as Hyper-V virtual machines with a Linux OS.

Q: Can Windows and Linux containers in the same environment talk to each other?

A: Yes! Windows and Linux containers in the same cluster can share a common overlay network.

Q: Do you support Windows Server 1709 or Windows Server 1803?

A: Support for Windows Server 1709 and Windows Server 1803 will be coming in a patch release in Q2 2018.

Q: How does licensing work for Docker EE 2.0 and Windows Server?

A: Docker EE 2.0 Basic (i.e. EE Engine) is included in Windows Server 2016, however, EE Standard and Advanced tiers, which provide the unified management UI and private registry capabilities, require additional licensing. Please contact sales for more information about pricing and licensing.

Q: What solutions are available for my .NET development team building containerized applications?

A: In addition to Docker for Mac, Docker also delivers Docker for Windows — a free desktop solution for local container development. There are two releases of Docker for Windows; the edge release includes a standalone Kubernetes server and client, as well as Docker CLI integration. The Kubernetes server runs locally within your Docker instance allowing your .NET developers to begin working with both Swarm and Kubernetes locally.

Original Link

Ready for Bionic Beaver? What’s New in Ubuntu 18.04

A couple of weeks ago I wrote a piece about how desktop Linux may be the only remaining “proper” desktop OS. The article proved popular and sparked some conversation around the topic in comments and social media surrounding it. With perfect timing, this week saw the release of the latest version of Ubuntu 18.04, and I joined two press calls to hear more details about this new LTS release of one of the most popular Linux distributions.

On the Desktop

Fittingly, the call kicked off with reiterating a commitment to Ubuntu desktop after years of experiments that ultimately failed. This step meant the abandonment of a handful of custom technologies and a switch back to more widely adopted open source projects such as Gnome Shell, perhaps providing a much-needed resource injection to these projects. Later in the call, there was a slight dig at some other hardware and software vendors, and the “artificial limits” they impose on users. Being free, Ubuntu lets you take things as far as you want.

Version 18.04, or “Bionic Beaver” (that’s quite a name) has a handful of interesting new features that reflect trends in the operating system world more widely:

Gnome Shell and Windowing Managers

Conscious of the disruption a major UI change can cause to users, 18.04 sets up Gnome Shell with default settings that are remnants of the Unity UI, such as keeping the application launcher on the left and retaining themes and colors.

After testing the Wayland in 17.10, the team found that while it’s increasingly stable, and is miles ahead of X.org in many ways, there were key issues (mostly related to screensharing) that resulted in a switch back to X.org for 18.04, upcoming releases will revert that change. Of course, being Linux, Wayland is still maintained and available if you want to use it instead.

A new community-contributed theme is also available, search for Communitheme (working title) in the snap store.

Live Updates

The new Canonical Livepatch Service lets you apply critical kernel security fixes without rebooting and reduces planned or unplanned downtime while maintaining security. It’s available as part of an Ubuntu Advantage subscription, or for all Ubuntu community members, it’s free for up to three machines. This feature keeps machines as up to date as possible with minimal effort and involvement.

Snaps

I wrote before about Canonical’s snap concept and efforts, and if the constant flurry of emails I receive from their PR team is anything to go buy, Canonical is working hard to grow the selection of applications. This selection now includes open and closed source applications, which will upset some members of the Linux community, but to be blunt, it’s a good strategy to bring users to Linux who may switch to open alternatives once they are comfortable with the platform.

Whether you like the concept or not, Canonical announced that the Snap store has seen over 1 million installs in the past 3 months alone, and the snap store is available for other Linux distributions.

Data Metrics

Always a controversial topic in the Linux community, but Canonical is insistent that tracking useful anonymous information about how people use Ubuntu and what hardware they use is important for them, and for developers creating applications for Ubuntu.

Ubuntu on Other Devices

As nice as some of these features are, Ubuntu’s primary user base is not on the desktop, but on a variety of other platforms (the majority of cloud providers use Ubuntu), so what does 18.04 offer for them? For developers, one of the significant advantages of using Ubuntu is common packages from development to these other platforms, ‘guaranteeing’ that they work.

Performance Optimization

Without giving much detail, Canonical has worked with all the cloud providers to roll-out optimized versions to their customers. Noting that with hybrid clouds the new normal, they improved inter-cloud communication and faster boot times for “bursty” applications and services. For an ultimate performance tweak, all 18.04 cloud images, OpenStack and Kubernetes distributions include support for hardware acceleration with NVIDIA GPUs. The OpenStack distribution also includes support for NFV and NVIDIA Tesla GPUs among others.

Adding to Intel and ARM architecture support, 18.04 also adds support for Power9 architectures suited to machine learning workloads.

Containers and Kubernetes

Nearly everyone is dipping their toes into Kubernetes, and Canonical optimized their own (mostly vanilla) blend for Google Cloud and specifically with AI and analytics tooling built-in. At a container level, Ubuntu 18.04 continues to have Docker and LXD support by default. If you’re not familiar with LXD, version 3.0 contains interesting new features, especially for those who need to maintain and manage outdated and insecure images without them affecting daily operations.

Like what you see? Read more and download 18.04.

Original Link

Will Linux (Finally) Win the Desktop War Because No One Else Cares?

Every year for as long as I remember Linux aficionados claimed that "next year will be the year of the Linux Desktop." This continued until sometime a couple of years ago. When much of the community realized that with the proliferation of servers, containers, and mobile devices (I’m claiming Android and ChromeOS to a degree) running Linux meant that the community had won the OS war anyway, just not on the desktop.

With Apple increasingly looking like it’s losing interest in macOS, and issuing mixed messages on the future of the platform, from promising to update its pro line, while simultaneously alluding to merge macOS and iOS hardware and software further. Read this Bloomberg article and you’ll realize that Apple’s homemade chips are actually pretty powerful and impressive, but they are proprietary, and only used by Apple.

Original Link

Starting MongoDB Database Software

In this post, we will cover how to start MongoDB database software in the three most used platforms: Windows, Linux, and MacOS.

If you have just started with NoSQL databases, you might wonder how to evaluate whether MongoDB is a good fit for your application.

Original Link

10 DevOps Interview Answers

In a previous post, I put together a list of 10 questions for DevOps team managers to use as a point of reference when interviewing potential DevOps candidates. Each question included an in-depth explanation and follow-up questions, so in case you were, or are now, a candidate, that list could help you prepare for interviews.

To help candidates even further, I’ve prepared this cheat sheet for those preparing for an interview for a DevOps position. Each organization and team is different of course, using different technologies and methodologies, but coupled with research on the organization you are interviewing at, these tips can be quite the resource.

What have you been doing over the last 1-2 years?

This is probably one of the best opportunities to let the potential employer know that you are a good fit for the role. You should practice answering this question, and make sure that your answer includes the proper terms of the industry, and that you know what they mean.

Make sure to provide information about your drive for automating things (explain why and give examples). Mention the tools you’ve used and be prepared to explain why you chose them. It is also a good idea to use this question to bring some of your personality traits to light: let them know that you took responsibility of something, mention how you excelled in one area or another, ideas you had, organizational changes you advocated, etc.

How do you deploy software?

At this point, you’re expected to go into the details of your deployment pipeline. It’s okay if the pipeline is not automated, but for each step, it’s a good idea to describe what is happening, what is good about it, and what are its weak spots, along with suggestions for improvement.

A bonus is to weigh in the benefits of each of these improvements along with the cost (in time and risk) of implementing them and to propose an effective roadmap for this critical part of work.

In certain parts of the interview, it is good to mention your use of source control, and why you are using it like that and not in another way. In general, it is very important to be able to answer “why” questions. Most of the time they are more important than the “what” questions.

Lastly, talk about your infrastructure changes, and how those are performed (hint: infrastructure as code).

How have you handled failed deployments?

First of all, it is important to demonstrate that you are not afraid of failures, that you understand that they are a part of the process and that failures need to be taken into account by reducing the time it takes to fail a component and having sound rollback processes in place in advance.

Talk about how much of this failure handling is done manually and how much is automated. It also shows professional maturity to have (and mention) a sound lesson learning function in your organization.

Lastly, don’t forget to mention the plan moving forward, when is it decided, and how the systems that are in place are conducive to that goal.

If something breaks in production, how do you know about it?

Talk monitoring, monitoring, and more monitoring. Both applicative (are my users able to log in? how long do my pages take to load? are there exceptions in my logs?) and resource usage/capacity.

If your environment is dynamic, talk about how your monitoring infrastructure is automatically modified to reflect your monitored environments. Do not forget to talk about escalations and customer communications.

What happens when you type “mv *” in a directory with three subdirectories: a, b, and c?

The answer is that the directories “a” and “b” are moved into “c.” The talking points are shell expansion, and understanding that “mv” can take any number of arguments, the last of which is the target.

The significance of this question is not in knowing the specific answer to it, but knowing enough about shells to not say silly things.

Without using Docker, can you see the processes running inside a container from the outside?

Yes! This is an opening for you to demonstrate your understanding of containerization fundamentals. ‘Docker-containerd-shim’ is the parent process of a process running inside a container, so it can be used for digging from the host OS, using simple ‘ps’ commands.

Docker is not a virtualization technology, but rather uses Linux building blocks such as cgroups and namespaces to provide isolation. As such, processes do not run on a separate (Guest) OS.

More useful information can be found in Jérôme Petazzoni’s talk on SlideShare or in his DockerCon EU 2015 talk on YouTube.

Describe the Linux boot process.

Needless to say, this question is aimed at gauging your system understanding and Linux expertise. I will not go into the whole explanation, but you can check out this resource for reference.

This, in a sense, is a very tricky question. More often than not, it does not require knowing the actual answer (although that would be a big plus), but it does require being able to talk 

How does “traceroute” work?

Traceroute sends multiple packets to the destination, with ever-increasing TTL (starting with 1), and listens for the incoming “time exceeded” responses coming in from all the intermediary routers.

More useful information on this is available here: http://users.cs.cf.ac.uk/Dave.Marshall/Internet/node77.html

Do you consider seven to be a high load average?

A question aimed at gauging your understanding of the basics of system performance monitoring. “Load” numbers are always relative to the number of CPU cores on the machine. So a plain “7” is meaningless. Could be low or high if the host has more than, or fewer than, 7 processor cores.

Even then it does not directly mean much and is indicative of the number of processes that are waiting for resources (these could be disk, network, CPU, etc.)

Do a FizzBuzz coding test.

It goes without saying, but I’ll add this anyway – it’s important your answer is correct:

  • Make sure your answer actually finds all the numbers along the way
  • Make sure it stops at the right moment
  • Make sure there are no glaring syntax errors
  • Make sure to catch all cases (a common mistake is that the “buzz” condition is never verified when “fizz” condition is met.)

It’s equally important how you behave while trying to solve this test. If you are blanking-out, try not to disrespect the question (this has happened to many candidates of ours before), complain or otherwise behave in a negative manner. Nobody wants to work with someone who can’t take a little adversity.

If you’re wondering why these questions are asked in the first case, I recommend referring to the previous article listing the questions. Suffice to say, organizations are finding it increasingly challenging to find talent and these questions are meant to separate the wheat from the chaff.

The fact that there is great demand for DevOps talent is the good news. The bad news is that there are many fantastic engineers out there competing for the same positions. So use this cheat sheet, prepare yourself, do your research.

As I mentioned in the previous article, I have conducted many, many DevOps engineer interviews. Each one has been different than the other. But what has made a difference between good candidates and excellent candidates is how solid their foundation was, and how sound their reasoning was for each question.

Lastly, and this probably goes without saying, we hire people who we would like to work with so being nice also helps.

Original Link

Oh, Git Configurations! Let’s Simplify It

Oh, Git Is Complex!

Git is a complex system. To see the proof, check out:

  1. ohshitgit.com
  2. gitfightrules.com

To mitigate that, we are going to get inside the head of Linus Torvalds! (Just a little bit – maybe into a single neuron). In this part, we will focus on configurations. I always thought that to understand a system, I needed to understand its configurations (or even better, its installation).

When I get to a new company, the first thing I try to figure out is:

  1. How do I install this?
  2. How do I configure this?

How Does Linus Torvalds Think?

“Everything is a file.”

Yes, this is how Linus thinks: everything is a file.

As Linus loves this idea that everything is a file, you can just view git configurations as a file.

So, if we manage to learn which files Linus uses in git, we might be able to penetrate oh-my-complex-git!

3 Fallback Layers of Config

  1. System – Your OS git config – git config --system
  2. Global – Your User git config – git config --global
  3. Local – Your Repository git config – git config --local
git config --system # => /etc/gitconfig git config --glboal # => ~/.gitconfig or ~/.config/git/config git config --local # => .git/config

Git first reads the git config from .git/config -> [fallback to] ~/.gitfconfig -> [fallback to] /etc/gitconfig.

Email per Repo

If you have different email addresses for different repositories, this goes into .gitconfig == git config --local

Get the Global Config With git config --list --global

➜ tmp git config --list --global user.name=Tomer Ben David
user.email=tomer.bendavid@ohohoh.com
core.autocrlf=input
format.pretty=format:%h %Cblue%ad%Creset %ae %Cgreen%s%Creset

Would that be the same as cat ~/.gitconfig? What do you think?

➜ tmp cat ~/.gitconfig
[user] name = Tomer Ben David email = tomer.bendavid@ohohoh.com
[core] autocrlf = input
[format] pretty = format:%h %Cblue%ad%Creset %ae %Cgreen%s%Creset

Yes and no! It’s a different notation, but generally the same!

Get the Merged Config With git config --list

The merged config is the combination of all configs with the hierarchy. Let’s see it on my machine:

git config --list ➜ tmp git config --list
core.excludesfile=~/.gitignore
core.legacyheaders=false
core.quotepath=false
mergetool.keepbackup=true
push.default=simple
color.ui=auto
color.interactive=auto
repack.usedeltabaseoffset=true
alias.s=status
alias.a=!git add . && git status
alias.au=!git add -u . && git status
alias.aa=!git add . && git add -u . && git status
alias.c=commit
alias.cm=commit -m
alias.ca=commit --amend
alias.ac=!git add . && git commit
alias.acm=!git add . && git commit -m
alias.l=log --graph --all --pretty=format:'%C(yellow)%h%C(cyan)%d%Creset %s %C(white)- %an, %ar%Creset'
alias.ll=log --stat --abbrev-commit
alias.lg=log --color --graph --pretty=format:'%C(bold white)%h%Creset -%C(bold green)%d%Creset %s %C(bold green)(%cr)%Creset %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
alias.llg=log --color --graph --pretty=format:'%C(bold white)%H %d%Creset%n%s%n%+b%C(bold blue)%an <%ae>%Creset %C(bold green)%cr (%ci)' --abbrev-commit
alias.d=diff
alias.master=checkout master
alias.spull=svn rebase
alias.spush=svn dcommit
alias.alias=!git config --list | grep 'alias\.' | sed 's/alias\.\([^=]*\)=\(.*\)/\ => /' | sort
include.path=~/.gitcinclude
include.path=.githubconfig
include.path=.gitcredential
diff.exif.textconv=exif
credential.helper=osxkeychain
user.name=Tomer Ben David
user.email=tomer.bendavid@ohohohoh.com
core.autocrlf=input
format.pretty=format:%h %Cblue%ad%Creset %ae %Cgreen%s%Creset

That was it – all the git config on my machine! I hope I didn’t put any passwords in there! If I did please, let me know – code review is important, guys!

Git Config –local/global/system user.name – Get a Single Config

Ask layer by layer for a config:

➜ tmp git config --local user.name
fatal: BUG: setup_git_env called without repository # We called from a non-repository folder.
➜ tmp git config --global user.name
Tomer Ben David
➜ tmp git config --system user.name # => nothing in the global config file.
➜ tmp

Aha! So it’s coming from the global (user) config!

Note that in the file, it’s with [user] and the name, and the git config returns the combined name user.name.

If you need a hierarchy larger than three, then in the file, it will look like this:

[branch "gke"] remote = origin merge = refs/heads/gke

So, this one should be branch.gke.remote. Let’s verify this:

➜ .git git:(gke) git config branch.gke.remote # => yes it is branch.gke.remote!
origin

Set a New Config With git config mysection.mykey myvalue

➜ .git git:(gke) git config mysection.mykey myvalue
➜ .git git:(gke) git config mysection.mykey
myvalue

So, we were able to set it. Let’s look at the file:

➜ .git git:(gke) cat config
[core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true ignorecase = true precomposeunicode = true
[remote "origin"] url = https://bitbucket.org/yshenbaum/k8s-course.git fetch = +refs/heads/*:refs/remotes/origin/*
[branch "gke"] remote = origin merge = refs/heads/gke
[mysection] ##### =====> Here it is! see the section in [] mykey = myvalue

Summary

You now know exactly where your git config files are. This is very helpful and much more explainable than using the git commands. I’m not sure I was able to simplify it, but it makes at least some sense at the end of the day for us programmers!

Resources

Ry’s Git Tutorial here is my BOOK Review

Original Link

Oracle Linux 7: ”One of the Configured Repositories Failed”

After a fresh install of Oracle Linux 7.4 and trying to run a ‘sudo yum update,’ I get:

One of the configured repositories failed (Latest Unbreakable Enterprise Linux Server 7 Server (x86_64)

Following the recommendations following the error didn’t help, and this post suggested trying a ‘sudo yum clean,’ but this didn’t fix it either.

Checking if I have network connectivity, it seems I can’t ping www.google.com or ping 8.8.8.8. I’ve come across this before with RHEL/CentOS installs and forgetting to enable the network interface options during install. Checking /etc/sysconfig/network-scripts/ifcfg-* the ONBOOT property was set to “no.” Changing it to “yes” and rebooting did the trick.

Original Link

Using the Turnkey Linux VM for Tracks Testing

I normally use VM Ware, but I create a video showing Turnkey Linux and Virtual Box to help people with the network settings.

Virtual Machines

Virtual Machines are great, particularly when someone else builds them for you, which is the case with Turnkey Linux.

For my book “Automating and Testing a REST API,” I used both Bitnami VMs and Turnkey VMs, but Bitnami stopped supporting the Tracks VM – Turnkey (at the time of writing) still support the Tracks VM.

I used Virtual Box and importing the Turnkey VM as a Virtual Box Appliance worked faultlessly.

The main issue that people have with VMs is the network settings.

VM Network Settings

With the Tracks VM, we don’t really care if the VM can access the outside world or not, we just want to access it from our browser and with our REST Client tools.

In the video, I work through all the Network settings to show the various errors that you might encounter, but normally, I use either

  • Bridged, or
  • Host-Only Adapter.

On VM Ware, I tend to use NAT.

Experiment With Different Settings

The point is, that with different VMs, on different operating systems, with different machines, and different Virtual Machine Host software, we have to experiment.

In the trade, this is known as “messing with the settings until it seems to work.”

I’ve uploaded the video to YouTube and embedded it on the support page for “Automating and Testing a REST API.”

Video

Original Link

Packaging a Node App for Docker from Windows

Container technologies are becoming a cornerstone of development and deployment in many software houses—including where I have my day job. Lately I’ve been creating a small web app with lots of vulnerabilities to use for security awareness training for developers (giving them target practice for typical web vulnerabilities). So I started thinking about the infrastructure: packing up the application in one or more containers, what are the security pitfalls? The plan was to look at that, but as it turned out, I struggled for some time just to get the thing running in a Docker container.

First of all, the app consists of three architectural components:

  • A MongoDB database. During prototyping I used a cloud version at mlab.com. That has worked flawlessly.
  • A Vue 2.0 based frontend (could be anything, none of the built-in vulnerabilities are Vue specific).
  • An Express backend primarily working as an API to reach the MongoDB (and a little sorting and such).

So, for packing things up, I started with taking the Express backend and wanting to add that to a container to run with Docker. In theory, the container game should work like this:

  1. Create your container image based on a verified image you can download from a repository, such as Docker Hub. For node applications the typical recommendation you will find in everything from Stack Overflow to personal blogs and even official doc pages from various projects is to start with a Node image from Docker Hub.
  2. Run your Docker image using the command docker run –p exposedIPhost:IP myimage  
  3. You should be good to access the running NodeJS app at localhost:hostIP

So, when we try this, it seems to run smoothly….until it doesn’t. The build crashes—what gives?

Image title

After some more googling we tried to use node:8-alpine as the base image. Didn’t work, it cannot install the necessary build tools to run libxmljs, with a warning that a required file is not available for the Alpine package manager.

Building on top of Alpine, a minimal Linux distribution popular for use in containers in order to reduce the image size, we try to install some OS specific build tools required in order to install the npm package libxmljs. This package is a wrapper for the xmllib2 library for C (part of the Gnome project). Because that is what it is, it needs to set up those bindings locally for the platform it is running on, hence it needs a C compiler and a version of Python 2.7 to make this happen. To install packages on Alpine one uses the apk package manager. These packages are obviously there, so why does it fail?

Normally building a NodeJS application for production would involve putting the package.json file on the production environment and running npm install. The actual JavaScript files are not transferred (stored on the folder node_modules), they are fetched from their sources. When installing modules that need to hook into platform specific resources, this is reflected in the contents of the local node module after first installation. So if you copy your node_modules folder over to the container, this can fail. In my case, it did: the app was developed on a Windows 10 computer, and we were trying to install it now on Alpine Linux in the container. The image was built with the local dev files copied to the app directory of the container image: and I had not told it what not to copy. Here’s the Dockerfile:

FROM mhart/alpine-node:8

WORKDIR /app
COPY. .

# Fixing dependencies for node-gyp / libxmljs
RUN apk add -no-cache make gcc g++ python

RUN npm install -production

EXPOSE 9000
CMD [“node”, “index.js”]

After adding the “no-cache” option on the apk command the libraries installed fine. But running the container still led to crash.

Image title

The error message we got when running the container was “Error loading shared library…. Exec format error”. This is because shared library calls are platform specific and built into the built version of libxmljs in node_modules.

After a few cups of coffee I found the culprit: I had copied the node_modules folder from my Windows working folder. Not a good idea. So, adding a.dockerignore file before building the image fixed it. That file includes this:

node_modules
backlog.log

The backlog file is just a debug log. After doing this, and building again: Success!

Now running the image with

docker run -p 9000:9000 -d my_image_name

gives us a running container that serves the Exposed port 9000 to the localhost port 9000. I can check this in my browser by going to localhost:9000

OK, so we’re up and running with the API. Next tasks will be to set up separate containers for the frontend and possibly for the database server—and to set up proper networking between them. Then we can look at how many configuration mistakes we have made, perhaps close a few, and be ready to start attacking the application (which is the whole purpose of this small project).

Original Link

Collecting Running Process Counts With Telegraf

Adding monitoring to your stack is one of the quickest ways to get visibility into your application, letting you catch issues more quickly and begin to make data-driven decisions about where to invest your engineering efforts. One of the most straightforward things to start monitoring is your processes themselves — it can be hard for a web server to serve requests if no processes are running, and on the flip side, you can quickly deplete your available resources by running more copies of a program than you intended.

Image title

On most ‘nix systems, process information can be gathered in multiple ways. Depending on which specific OS you’re running, you might be able to look at the proc filesystem, which contains a number of files with information about running processes and the system in general, or you could use a tool like ps, which outputs information about running processes to the command line.

In this example, we’ll use Python and Ubuntu Linux, but many of the concepts will carry over to other languages or applications and operating systems.

Getting Info About Processes on Linux

One great place to get more information about your system is the proc filesystem, which, according to the man page, “is a pseudo-filesystem which provides an interface to kernel data structures. It is commonly mounted at /proc.” If you visit that directory on a Linux machine, you might see something like this (output from a fresh install of Ubuntu 16.04.3):

$ cd /proc
$ ls
1 1182 14 18 25 3 399 437 53 60 80 839 bus driver kallsyms locks partitions sys version_signature
10 12 141 19 26 30 4 47 54 61 81 840 cgroups execdomains kcore mdstat sched_debug sysrq-trigger vmallocinfo
1017 1200 15 2 266 31 413 48 544 62 817 890 cmdline fb keys meminfo schedstat sysvipc vmstat
1032 1230 152 20 267 320 414 480 55 66 818 9 consoles filesystems key-users misc scsi thread-self zoneinfo
1033 1231 16 21 277 321 420 49 56 671 820 919 cpuinfo fs kmsg modules self timer_list
1095 1243 164 22 278 369 421 5 57 7 828 925 crypto interrupts kpagecgroup mounts slabinfo timer_stats
11 126 165 23 28 373 423 50 58 701 831 acpi devices iomem kpagecount mtrr softirqs tty
1174 128 166 24 29 381 425 51 59 79 837 asound diskstats ioports kpageflags net stat uptime
1176 13 17 241 295 397 426 52 6 8 838 buddyinfo dma irq loadavg pagetypeinfo swaps version

There’s a lot there! The first thing you’ll notice is a series of numbered directories; these correspond to running processes, and each directory is named with the Process ID, or PID, of that process. You’ll also see a number of other directories and files with information about everything from kernel parameters and loaded modules to CPU info, network statistics, and system uptime. Inside the directories for each process, you’ll find almost as much information about each individual process — too much for our use case. After all, we just want to monitor whether the process is running, and maybe how many copies are running.

When a system administrator logs into a server to verify that a process is running, it’s unlikely that /proc would be the first place they turn. Instead, they’d probably use a tool like ps, which also provides information about running processes. There are a few different versions of ps that you might use, but for the version on Ubuntu, you can use the following command to get information about all running processes on your server:

$ ps aux

We’re going to use Python to create a few processes for testing purposes. Since we don’t really need these to be doing any work, we’ll write a simple program with an infinite loop and a call to the sleep function in Python’s time module, in order to avoid using unnecessary CPU cycles. Make sure you have Python installed by entering the following command at the command line:

$ python --version Python 2.7.12

Since our program is so simple, it will work with either Python 2 or Python 3. Next, use a text editor to create a file called loop.py with the following contents:

#!/usr/bin/env python import time while True: time.sleep(5)

The first line tells the operating system which interpreter to use to execute the script. If this program was more complex or used functionality that differed between Python 2 and Python 3, we’d want to specify which version of Python we were using instead of just saying python.

Run this command from the same directory where the file is located to make the script executable:

$ chmod 744 loop.py

And then run the program, appending the & character to the end of the command (which tells Linux to run the process in the background) so we still have access to the shell:

$ ./loop.py &
[1] 1886

After running a command using the & character, the PID of the running process is listed in the output. If you run ps aux again, you should now see a Python process with PID 1886 in the list of results.

On the Ubuntu server I am using, this command returned just over 100 results, and searching through that list manually is too inefficient. We can use another command, grep, as well as some of Linux’s built-in functionality to narrow down the results. The grep command acts like a search filter, and we can use a Linux “Pipe”, the | character, to send the data from the output of the ps aux command to the input of the grep command. Let’s try it out:

$ ps aux | grep python
noah 1886 0.0 0.2 24512 6000 pts/0 S 20:14 0:00 python ./loop.py
noah 1960 0.0 0.0 14224 1084 pts/0 S+ 20:56 0:00 grep --color=auto python

First, we’re getting information about all running processes. Then, we’re “piping” that data into the grep command, which is searching for any lines that contain the string python. Sure enough, there is our Python process, 1886, in the first line of the results. But what about that second line?

When we run the ps command, the output includes the arguments we provided when each process was started; in this case, --color=auto is added because Ubuntu has an alias that runs grep --color=auto when you type grep, and then the python argument, which is the string we were searching for. So we’re searching for “python”, which means the string “python” will be included in the output of ps for the grep process, so grep will always match with itself because it contains the string it is searching for.

A common workaround to this issue is to search for the regular expression “[p]ython” instead of the string “python”. This will cause grep to match any string that starts with any of the letters inside the brackets, in our case only a “p”, followed by the letters “ython”. When we do this, grep still matches the word “python”, because it starts with a “p” and ends in “ython”, but it does not match itself because “[p]ython” doesn’t match that pattern. Give it a shot:

$ ps aux | grep [p]ython
noah 1886 0.0 0.2 24512 6000 pts/0 S 20:14 0:00 python ./loop.py

Let’s start up another Python process and see what we get:

$ ./loop.py &
[2] 1978
$ ps aux | grep [p]ython
noah 1886 0.0 0.2 24512 6000 pts/0 S 20:14 0:00 python ./loop.py
noah 1978 0.0 0.2 24512 6004 pts/0 S 21:13 0:00 python ./loop.py

Two Python processes, two results. If we wanted to verify that a certain number of processes were running, we should just be able to count the lines outputted by our command; fortunately providing the -c argument to grep does exactly that:

$ ps aux | grep -c [p]ython
2

Let’s bring the most recent of the two Python scripts into the foreground by using the fg command, and then kill it using Ctrl C, and count the number of Python processes again:

$ fg
./loop.py
^CTraceback (most recent call last): File "./loop.py", line 6, in time.sleep(5)
KeyboardInterrupt
$ ps aux | grep -c [p]ython
1

Perfect! 1 is the number we were looking for.

There’s another command, pgrep, which also fulfills all the requirements of our use case, but it’s not as generally useful. It allows you to search for all processes which match a string and returns their PIDs by default. It also accepts a -c argument, which outputs a count of the number of matches instead:

$ pgrep -c python
1

Gathering Process Counts With Telegraf

Now that we know how to count the number of processes running on a server, we need to start collecting that data at regular intervals. Telegraf gives us a way to execute the same commands that we’re using at the shell in order to collect data in the form of the exec input plugin.

The exec plugin will run once during each of Telegraf’s collection intervals, executing the commands from your configuration file and collecting their output. The output can be in a variety of formats, including any of the supported Input Formats, which means that if you already have scripts that output some kind of metrics data in JSON or another of the supported formats, you can use the exec plugin to quickly start collecting those metrics using Telegraf.

If you don’t already have Telegraf installed, you can refer to the installation documentation here. After following the instructions for Ubuntu, you should find a config file located at /etc/telegraf/telegraf.conf.

For the purpose of this example, we’re going to write the output to a file, so we want to edit the [[outputs.file]] section of the config like so:

# # Send telegraf metrics to file(s)
[[outputs.file]] ## Files to write to, "stdout" is a specially handled file. files = ["/tmp/metrics.out"] ## Data format to output. data_format = "influx"

We’ll apply those changes by restarting Telegraf, then check that metrics are being written to /tmp/metrics.out. When installing Telegraf from the package manager, the system input plugin is enabled by default, so we should start seeing metrics immediately:

$ sudo systemctl restart telegraf
$ tail -n 2 /tmp/metrics.out
diskio,name=dm-0,host=demo writes=7513i,read_bytes=422806528i,write_bytes=335978496i,write_time=23128i,io_time=9828i,iops_in_progress=0i,reads=9111i,read_time=23216i,weighted_io_time=46344i 1519701100000000000
diskio,name=dm-1,host=demo write_time=0i,io_time=108i,weighted_io_time=116i,read_time=116i,writes=0i,read_bytes=3342336i,write_bytes=0i,iops_in_progress=0i,reads=137i 1519701100000000000

Unfortunately, the exec plugin doesn’t know what to do with multiple commands, like we have above, so we need to put them into a simple bash script. First, create a file called pyprocess_count in your home directory with the following text:

#!/bin/sh count=$(ps aux | grep -c [p]ython) echo $count

This script serves a secondary objective besides allowing us to execute a piped command using the exec plugin- if grep -c returns zero results, it exits with a status code of 1, indicating an error. This causes Telegraf to ignore the output of the command, and emit its own error. By storing the results of the command in the count variable and then outputting it using echo, we can make sure that the script exits with a status code of 0. Be careful not to include “python” in the filename, or grep will match with that string when the script is run. Once you’ve created the file, set its permissions so that anyone can execute it and test it out:

$ chmod 755 pyprocess_count
$ ./pyprocess_count

Then move it to /usr/local/bin:

$ sudo mv pyprocess_count /usr/local/bin

Next, we need to configure the exec input plugin to execute the script. Edit the [[inputs.exec]] file so it looks like this:

# # Read metrics from one or more commands that can output to stdout
[[inputs.exec]] ## Commands array commands = [ "/usr/bin/local/pyprocess_count" ] ## Timeout for each command to complete. timeout = "5s" name_override = "python_processes" ## Data format to consume. ## Each data format has its own unique set of configuration options, read ## more about them here: ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md data_format = "value"

We’ve added the command directly to the command array, so it will be executed by Telegraf once per collection interval. We’ve also set the data_format to "value" because the command will output a single number, and we use name_override to give the metric a name.

Restart Telegraf again and then look at the metrics.out file to see if our new metrics are showing up. Instead of searching through the file by eye, we can use grep again to search for any lines with “python” in them:

$ grep python < /tmp/metrics.out
python_processes,host=demo value=1i 1519703250000000000
python_processes,host=demo value=1i 1519703260000000000
python_processes,host=demo value=1i 1519703270000000000

We’re using the < character to send the contents of the metrics file to the grep command, another Linux feature, and in return, we get a few lines of metrics in InfluxDB line protocol with the name of the metric, a tag for the host added by Telegraf, the value (with an “i” to indicate that it is an integer), and a timestamp.

If we bring up another Python process, we should see the value change in our output:

$ ./loop.py &
[2] 2468
$ grep python < /tmp/metrics.out
python_processes,host=demo value=1i 1519703250000000000
python_processes,host=demo value=1i 1519703260000000000
python_processes,host=demo value=1i 1519703270000000000
python_processes,host=demo value=1i 1519703280000000000
python_processes,host=demo value=1i 1519703290000000000
python_processes,host=demo value=2i 1519703300000000000
Copy

And there we go! The final metric shows two Python processes running.

Next Steps

Writing metrics to disk isn’t a very useful practice, but it’s good for making sure your setup is collecting the data you expected. In order to make it actionable, you’ll need to send the data you collect to a central store somewhere so that you can visualize and alert on it.

The visualizations for these metrics would be minimal; we probably don’t need a full graph since there shouldn’t be much variation in the data we’re getting that we need to look at historically. Instead, displaying a single number (for example, the Single Stat panel in Chronograf) should be enough to give you some confidence that things are working as expected.

How you alert on these metrics will depend on what exactly you’re monitoring. Perhaps you always want to have one copy of a process running. You could create an alert that sends an email every time your process count dropped below 1. After the first few alerts, though, your team will probably want to automate bringing up a new process if yours crashes, so you’ll need to tweak the alert so that some time needs to elapse between seeing the metric go to 0 and sending the first alert; if your automated system can bring up the process quickly, then a human doesn’t need to be contacted.

Or maybe you have a system that is regularly spawning new processes and killing old ones, but which should never have more than X processes running at a given time. You’d probably want to set up a similar alert to the one above, except instead of alerting when the metric drops from 0 to 1, you’d alert if the metric was greater than or less than X. You might want to give yourself a time window for this alert as well; maybe it’s OK if your system runs X+1 or X-1 processes for a short time as it is killing and bringing up new ones.

If you decide to send your data to InfluxDB, you can use Chronograf and Kapacitor to visualize and alert on your metrics. You can read more about creating a Chronograf Dashboard or setting up a Kapacitor alert on their respective documentation pages.

Original Link

Which Distribution of Linux Should I Use?

I’m often asked this question: “hey, you’re a Linux guy right? What Linux should I use? I have this friend who recommends _____ and I want to know what you think?” I usually reply with the same question: what do you want to do? So I decided to make a blog post about it that I can send people instead.

My History With Linux

I should probably preface this article with a little bit of my history with Linux, in case you’re reading this and you don’t know me (very likely). You can probably skip this if you don’t care.

I started out using Linux around 1996. My first Linux was Slackware 4, a set of CDs I purchased at Egghead Software (yep, I’m old). A friend of mine told me about this Unix like thing that was so great and I just had to try it and he thought I would love it.

I read a lot about Unix, and was very curious about it. I had a shell account at my internet provider and I’d tinkered around, yet at first I was a bit hesitant. “Why would I need this?” His reply was simply: “Because you hate Windows 95 so much and love DOS, you’ll love this”. So I bought it. He was right.

I took an old hard drive I had and installed it. I fought with it for hours, then days. I finally got a desktop up and running. I have no idea what drove me in this time, but I had to figure out how to make this system work, and it was difficult. I had to know so much about my hardware! Simple things were suddenly hard again. But I pushed through, and I got my desktop up. And I started building some silly scripts for fun. The system was fast, and I could change nearly everything about it.

It had a built in C compiler? I just bought some really expensive Borland package for this I could barely figure out. But this OS had a compiler built in? A free Image editor? I was hooked!

For years after that, I experimented with tons of Distributions. Even BSD Unix ones. My “main computer” was always a dual boot, and some of them were pure Linux. Most of the early 2000s, I avoided Windows completely. So by year, I can break it down to my “main machine”, it would be:

  • 1996-1999: Slackware
  • 1999-2002: Redhat (and FreeBSD)
  • 2003-2005: FreeBSD / Knoppix
  • 2005-2009: Gentoo
  • 2009-2011: Linux Mint
  • 2011-2018: Arch Linux / Debian

So, of course, I used probably 50 or more distributions in my time, but this was what was running on my “main machine” I used for work, or browsing, or development or whatever. Obviously Arch had the longest run so far, mainly because I could just configure it and forget about it for long periods of time.

But the main distro for my “real work” the last few years has been Debian.

Enough about me though, let’s talk about what you should use.

So What Do You Want to Do?

I’m going to put these in categories based on common needs. There is some overlap here, and with enough work any of these Linux distributions will work for your desired needs. One of the great things about Linux is you can make it whatever you want. But some distributions do a lot of that work for you, or have a design that works better towards certain goals. I’ll present these in categories based on the easiest path to reach your goals.

I’m a Linux Newbie Just Getting Started

For a long time I recommended Ubuntu for this. As far as ease of use and compatibility it was great. But I pretty much hate Ubuntu now. I still use it for demos in my courses and articles because so many people use it, but I am not a fan of the way they run this distribution, the built in Amazon adware, and Unity is annoying.

So if you’re just starting out I recommend:

It’s kind of a cheat because Linux Mint is built off Debian, but Mint looks prettier and has some nice cross-platform stuff.

Use these distributions if you want:

  • A Windows-like experience
  • Something simple to install
  • Something reliable
  • Something “Linux like” that doesn’t deviate from the norm
  • Something that “just works”

Okay, so that last one is really important. It just works. Either of these distributions are plug and play. Set them up, and forget about it. I have become increasingly reliant on Debian for my development machines because at times I don’t care about the OS and I don’t want it to get in my way. When I’m in a mood where I just want to build things, it can’t be beat.

I Want to Learn More About Linux/Unix and My Hardware

Maybe you’re in the mood to play and experiment. You want to challenge yourself and force yourself to learn by doing. That’s great, it’s exactly what I did.

If you want to challenge yourself and learn, I recommend:

Each of these distributions requires a lot of configuration, hardware discovery, and compiling of source. With Gentoo you have to compile everything. It’s a great way to have absolute full control over your operating system.

Use these distributions if you want:

  • Full control of your computer and OS
  • To learn about Linux internals
  • A lean and mean optimized system

This comes at a cost: mainly your time. A full install of these can take hours. On the plus side, they tend to run forever.

I had an Arch Install on a Lenovo that took the better part of a Saturday to configure, and let’s say another 10 hours or more spread out after that. It ran nearly effortlessly for 5 years (till the laptop hardware died). I only had to do a few updates once in a while, but I used it reliably every day for 5. Long. Years. So in a way you can look at it as investment.

I Want Cutting Edge Stuff

Okay, maybe you want the latest greatest software and you don’t care how stable it is. You want to do some kernel hacking or some other cool thing that some coder committed yesterday.

To hell with stability and security, you want the newest thing now.

Use these distributions if you want:

  • To trade risk for the newest stuff
  • The latest and greatest features always
  • Fun configuring things to work with breaking changes

To be fair I’ve personally used Arch and Gentoo without significant stability problems, but I was risking using the bleeding edge stuff on rolling releases.

I Just Want to Get Some Work Done

Ok maybe you don’t really care about the OS particulars and just want to GSD (Get Stuff Done). Maybe you have some Node or GoLang apps you want to build and heard Linux is the best for it.

These are great for getting work done:

Use these distributions if you want:

  • Smooth operation with low maintenance
  • Minimal configuration
  • Things that just work mostly automatically
  • Compatibility with hardware and software

As I said I often use Debian these days as I’m usually just making something and don’t really feel like tinkering around and optimizing. It’s stable, fast, and stays out of my way. I’m writing this article in Debian 9 right now.

I Want to Set Up a Server

Maybe you want to set up a web server or virtual host and don’t know what to use. The first one on this list is the dominant distribution for web hosting, so if you want something that mimics the site that’s hosting your software, try CentOS (or learn Docker!)

These are solid and reliable for web hosting:

Use these distributions if you want:

  • Stability
  • Security
  • Support from other people using it for the same reason

I believe any Linux distribution can be used for web hosting effectively, but some take more work than others.

I Want the Most Performance Possible

So if you’re one of those types who wants to squeeze out every ounce of performance (I’ve been there) these are great for you. Some of these require compiling all the source code to produce binaries optimized for your processor(s). Fun stuff!

Use these distributions if you want:

  • Fast performance
  • High Load Computing

Keep in mind that hardware has reached a performance point where these don’t matter quite as much as they used to. 15 years ago you could hack a kernel and dial in your services and see a big boost. These days, the difference is negligible. Any Linux will be pretty snappy.

I Want a Secure Desktop

Maybe you want to set up a system that’s hard to break into, for whatever reason. There are a couple distributions with security as a top focus. If you’re really concerned about locking down your main machine, these are great ones to look at.

Use these distributions if you want:

  • Security

  • Anonymity

I Want a Minimal Computer System

Ok sometimes you just want something lean and mean that gets a certain job done. I definitely understand this. Maybe you have an old Pentium you want to re-purpose. Sometimes the OS is just a small part of your goal and you want the bare minimum.

Use these distributions if you want:

  • Something that will run on old hardware
  • Something minimal as possible

Conclusion

I hate to sound like a broken record, but you could just pick out one of these Linux distributions and make it whatever you want. That’s the nature of Linux, its customizable to the furthest degree. But these are great distributions for getting started fast. If you think I’ve missed the mark or left out a distribution feel free to leave me a message in the comments, or yell at me on Twitter.

And whatever you do, if you reached this page because you’re curious about Linux, try it out!! Now! These days you can download something like VirtualBox (free of charge) and try it out before really committing to anything. It’s definitely worth your time to check it out!

Original Link

What Open Source Software Do You Use?

To gather insights on the current and future state of open source software (OSS), we talked to 31 executives. This is nearly double the number we speak to for a research guide and believe this reiterates the popularity of, acceptance of, and demand for OSS.

We began by asking, “What Open Source software do you use?” As you would expect, most respondents are using several versions of open source software. Here’s what they told us:

Apache

  • Apache Cassandra, Elassandra (ElasticSearch + Cassandra), Spark, and Kafka (as the core tech we provide through our managed service) are the big ones for us. We find that the governance arrangements and independence of the Apache Foundation make a great foundation for strong open source projects.
  • 95% of what we do with big data is open source. We use Apache Hadoop and contribute back to grow skills and expertise.
  • We use so much that it would be impossible to list. The core of our software is based on Apache Solr and Apache Spark. We contribute to Solr extensively.
  • Apache license project. Building blocks Apache Arrow started and led. In memory columnar data. Apache Calcite SQL processing engine used by Hive. Apache Parquet store columnar data on disk. Other projects to streamline engineering efforts. Frontend JavaScript React. Apache Zookeeper coordinates different states. RocksDB from Facebook to manage metadata.
  • We use several projects from the Apache Software Foundation within its own operations, such as Apache CXF, Apache Karaf, Apache Camel, Apache ActiveMQ, and Apache Beam. We also use Eclipse projects like ECP.
  • We are directly contributing to Apache Flink and Apache Beam. We work with a plethora of projects in the ecosystem, like Kafka, the Hadoop stack, the Scala ecosystem, etc. For our product, the dA Platform 2 (an out-of-the-box stream processing platform) we are building heavily on Kubernetes.
  • The data processing layer of Magellan is based on Apache Spark and runs on Hadoop, both household names in Open Source. 
  • We contribute to open source projects, but it is also a big consumer of open source software. Many of our products use common open source libraries from Apache Foundation. We enable integrations with a range of ecosystems which have significant open source tooling built around them. We use open source software from a range of major companies – amongst them are Apache, Linux, and Eclipse foundations. Chief platforms used by us are Apache Commons, Maven, Eclipse, and Ant.
  • We use many different open source software components when building real-time streaming analytics applications for our customers. Some examples of open source software we use include Apache Apex for real-time streaming, Drools for adaptive rules, Druid for OLAP, Apache Kafka for message transport and Apache Spark for machine learning.

Linux

  • Internally, we use compilers such as GCC and Clang to build software, Linux as our development and testing environment, tools such as Docker and Kubernetes to make it easy to deploy the database. We use Nginx as a load balancer, and Apache Spark to analyze data. All of these are open source software.
  • We use Linux to build and run. Use Linux kernel APIs for performance. We also use the GCC compiler and C++17.
  • We use and produce open source products. Our solutions are based on Linux, Docker, Kubernetes. Zenko is our new product and is open source for petabytes of storage.
  • That’s a hard question because there isn’t any one open source ecosystem. The open source ecosystem that gives us the Linux operating system is quite different from the open source ecosystem that works on statistical analysis and the R programming language, which is again different from the open source ecosystem around JavaScript. 
  • We use open source software at all levels. At the core, we use Selenium and Appium for the actual automating that we do. The service above that is written in Python and runs on Linux, and so is open in those ways. 
  • We use OSS in our backend and frontend software components, application servers, programming languages, databases, testing tools and operating systems. Linux, components from the Apache Software Foundation, Glassfish, Node & NPM, and JavaScript components like JQuery are just some of the hundreds of Open Source projects we depend on.

Kubernetes

  • Open source frameworks such as OpenStack and Kubernetes, Puppet, Chef, Ansible, Jenkins, and others.

Docker

  • Not applicable as we are a creator of Open Source software; however, our product is reliant upon Docker, which is based on the Moby open-source software.
  • Integrate and manage Docker, MongoDB, and the .Net framework. We use Linux software from Red Hat to build and tune our Linux product. We use open SSL to encrypt messages.

Other

  • Not really relevant to us as we see all types. It’s more a question of time and are you staying current with releases. If 90 to 95 percent of code is open source and you “set it and forget it,” you’re only keeping five to 10% of your application up to date, and this is why more than 75% of applications have vulnerabilities.
  • Project Flogo and the technology behind it in itself is open source. We also leverage things like Kafka, AWS’ open source SDKs, TensorFlow, and a number of different open source HTTP style interfaces.
  • C++
  • We support all open source software around our four areas of focus: cloud, data, AI, and transactions.
  • GNU Step cross-platform library to which we also make significant contributions. Web tools based on the Ruby stack and we contribute Ruby Gems. IDP library for remote access. TensorFlow library for the AI side.
  • At the moment, we use around 50 different libraries, most of them from the Java ecosystem. 
  • Erlang OTB for cluster management for memory management and IO in C and C++. Nickle query is written in Go. Full-text search capability – created Bloody widely used full-text search library written in Go.
  • Our commercial software is built upon a solid foundation of open source software like PostgreSQL, ElasticSearch, nginx, and our user interfaces use Angular, Node.JS, Typescript, and many other components. And of course, we use our own open source projects like Chef, InSpec, and Habitat to build, deploy and manage both our on-premise and cloud offerings.
  • All of the projects we work with at are published using an Open Source Initiative (OSI)-approved license. As a member of the community, we recognize using an open source license is not enough – successful open source projects must also be open for collaboration. We are proud to work with the Open Source Geospatial Foundation, LocationTech, OGC, and other open source incubation foundations that address this need. 
  • We support integration and the following open source ecosystems and platforms: 1) Selenium for web application test automation; 2) Jenkins for software development and delivery automation; 3) MQTT and CoAP tooling for IoT testing; 4) Cucumber and SpecFlow for Behavior Driven Development processes; and, 5) IDEs including IntelliJ and Eclipse.
  • We use a wide variety of different OSS tools in our stack. We build our software with open source languages and compilers such as C (via GCC and Clang), Go, Ruby, Python, Perl, and Rust. We use Linux on our servers and much of our networking hardware. Inside our software, we use a huge variety of open source software, from the most popular frameworks down to small libraries like libfsm. Even our testing strategies involve open source software such as Theft. Finally, the core of our product is Varnish, a well-known open source reverse proxy server.
  • We are deeply committed to WordPress. We contribute time, code, support, and are deeply involved in the passionate WordPress community. Our top priority is providing the best-managed WordPress experience available, and that means using whatever tools necessary to make WordPress run and scale smoothly.  We’ve got firm roots in the LAMP stack, and we’ve invested heavily in OpenStack, Ceph, and Kubernetes, too!

Here’s who shared their insights with us:

Original Link

How to Run Payara Micro on Android (No Root Required)

Have you ever wondered, how Micro is Payara Micro?

At first, this was a fun experiment installing a terminal app on my Android phone and playing around with some Linux commands. One thing lead to another and I ended up running a Payara Micro instance on my device! In this article, I’m going to show you exactly how to do that.

For convenience’s sake, you may want to run this experiment from the comfort of your PC by connecting it via SSH to your device.

Installing Termux and a Linux Distribution

The best terminal app I found is Termux, which is an open source terminal emulator that provides a Linux environment. This means that you can install packages, run common Linux commands, and even configure startup processes when you boot up your phone. One of the most amazing things you can do with it is install community-made images of Linux’s distributions like Arch, Debian, Fedora, Kali, Ubuntu, etc.

However, even with Termux, you can’t install a JDK yet. After several tries, I installed Fedora 27 in Termux since I realized that is the most lightweight distribution for this requirement. To install it, just type the following commands in Termux (use f27_armor f27_arm64 depending your device):

pkg install wget
wget https://raw.githubusercontent.com/nmilosev/termux-fedora/master/termux-fedora.sh
bash termux-fedora.sh f27_arm

This operation will take from 10 to 30 minutes depending on your device, so be patient. After the installation is complete start the Fedora environment with the startfedora command:

1 - Payara Micro on Android.png

2 - Payara Micro on Android.png

Installing OpenJDK 8

Now that we are inside of our Fedora 27 environment, we are going to install OpenJDK v8 using the YUM package manager:

yum install java-1.0.8-openjdk

Fedora might want to update some dependencies before installing the JDK, so this might take a while. If you check the Java version with java -version after installing the JDK, you will notice that it will fail; don’t worry about it since it doesn’t affect our goal to run an instance of Payara Micro. 

Downloading Payara Micro and Deploying an Application

Let’s download the Payara Micro JAR file. To do this, just copy the distribution link from the Payara website and use it with the following command:

wget "<PAYARA_MICRO_DOWNLOAD_URL>" -O payara-micro.jar

It’s recommended to also download a WAR file like this one to test the server.

This WAR file is a modified project of my JavaEE 7 / Gradle starter project with just an index.html page and 3 sample RESTful web services:

<host>:<port>/api
<host>:<port>/api/list
<host>:<port>/api/query?name=&age=

Now that we have a JDK installed, the payara-micro JAR file and a sample WAR artifact, it’s deployment time!

java -jar payara-micro.jar --deploy Application.war

4 - Payara Micro on Android.png

9 - Payara Micro on Android.png

This may take several minutes depending on your device, and may throw some exceptions (in some cases), but your application will be deployed nonetheless.

In this case, the server is listening on port 8080 by default, so go to the device web browser and launch the http://localhost:8080 website: 

5 - Payara Micro on Android.png

6 - Payara Micro on Android.png

7 - Payara Micro on Android.png

Open the Payara Micro Server Outside Your Local Network

In case you want to show your new portable Payara Server to your friends, you can install a service called Ngrok

To do this, you need to download the ARM version and install it on your Termux terminal. Once installed you can open your server with the following command:

./ngrok http 8080

-which will give you an URL to see your server from the Internet. Isn’t it cool?

8 - Payara Micro on Android.png

Final Notes

At this time, the steps to achieve all of the above are a bit convoluted, so maybe in the future, the entire process can be simplified to ease running a Payara Micro instance on your smartphone. 

Remember that all of this depends on your device specifications, which means that the server can run too slow or become unresponsive. Using my Nexus 5X smartphone, it took up to 5 minutes for the Payara Server’s homepage to load, but on a separate tablet with more RAM the page loaded faster.

With an ever increasing performance in mobile processors each year, maybe we can start thinking about creating a distributed-mobile fleet of Payara Micro Servers that talks to several IoT devices using an array of microservices? Can you imagine that?!

Nearby devices sharing real-time data powered by Java EE and/or MicroProfile? Future possibilities are unlimited…!

Original Link

Installing Java 8 on AWS EC2 Amazon Linux instance

Today we are going to install Java 8 (JDK 8, I must say from the developer’s perspective) in an Amazon EC2 instance which is having an Amazon Linux version installed on it. In a nutshell, we can say that Amazon Linux can be considered equivalent to CentOS 6.

I am not going to waste your time and will instead jump directly towards the commands that you have to execute to install JDK 8 on your new instance. Before we start, please ensure you have the administrative privileges. OK. So, you have to execute the following commands as mentioned below.

Installing Java 8

$ cd /opt $ sudo wget --no-cookies --no-check-certificate --header "Cookie: %3A%2F%2Fwww.oracle.com%2F; -securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz $ sudo tar xzf jdk-8u151-linux-x64.tar.gz $ cd jdk1.8.0_151/ $ sudo alternatives --install /usr/bin/java java /opt/jdk1.8.0_151/bin/java 2 $ sudo alternatives --config java There are 2 programs which provide 'java'. Selection Command
----------------------------------------------- *+ 1 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java 2 /opt/jdk1.8.0_151/bin/java Enter to keep the current selection[+], or type selection number: 2 

(If you are getting some other options as well, choose the one with “/opt/jdk1.8.0_151/bin/java“)

Now let’s set alternatives, so that once Java is called, it points to the correct JDK version through following commands:

$ sudo alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_151/bin/jar 2 $ sudoalternatives --set jar /opt/jdk1.8.0_151/bin/jar $ sudo alternatives --set javac /opt/jdk1.8.0_151/bin/javac

Check Java Version 

$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Setting Environment Variables.

We can set the environment variables either user based or globally. With following approach, we are setting the environment variable globally, so that any user on this machine can access it. 

# vim /etc/profile export JAVA_HOME=/opt/jdk1.8.0_151
export JRE_HOME=/opt/jdk1.8.0_151/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin Esc + :wq! (To save file)

Congratulations!! You did it!

Original Link

Dad, Are You Using Linux? [Comic]

Download the blueprint that can take a company of any maturity level all the way up to enterprise-scale continuous delivery using a combination of Automic Release Automation, Automic’s 20+ years of business automation experience, and the proven tools and practices the company is already leveraging.

Download the ‘Practical Blueprint to Continuous Delivery’ to learn how Automic Release Automation can help you begin or continue your company’s digital transformation.

Topics:

comic ,golang ,linux ,devops

Original Link

How to Create a Database Backup Scheduler Script

Our goal in this article is to configure a script that will keep only the last seven days of database backup files.

Let’s get started.

First, create a bash script file backupexecutor.sh where we will write all our backup scripts.

Then, write an expect script file backup.exp where we will execute backupexecutor.sh and provide a password to execute backupexecutor.sh. We also write our output log script in our backup.exp file.

To do our database backup, we need only two files:

  1. backupexecutor.sh

  2. backup.exp

So, your Linux system must have expect installed.

To check if expect installed, type:

$ which expect

This will give an output like this:

/usr/bin/expect

If your system can’t recognize expect, then install it in your system by typing:

apt-get install expect

Or if you’re using Amazon Linux, it’s CentOS-based (which is RedHat-based), so type:

yum install expect

Now, you are ready to write your backup script.

backupexecutor.sh:

#First get current date and time for logging
#and to use as suffix in our Database backup file name
#get date yyyy-MM-dd
NOW_DATE=$(date +%F) #get time hh:mm:ss
NOW_TIME=$(date +%T) #current date time
NOW="$NOW_DATE($NOW_TIME)" #To keep last few days backup files only
#backup date
BACKUP_DATE=7 #Location of your database backup file.
#In this location your database backup file will generate
#backup location
LOCATION=/home/user/DB_BACKUP/BACKUP_FILES #Database user name
DB_USER_NAME=root #Database Name
DB_NAME=sampledb #Your database backup file name
#Example: If database backup executed on this date time 2017-11-08(20:30:01)
#Database backup file name will be myDatabase_2017-11-08(20:30:01).backup
#file name
FILE_NAME=myDatabase_$NOW.backup echo "==================================== Start $NOW ===================================="
echo "#Starting to take Database backup......"
#For postgresql
pg_dump -U $DB_USER_NAME -h 127.0.0.1 -W -F t $DB_NAME > $LOCATION/$FILE_NAME
#For mysql (uncomment below)
#mysqldump -u $DB_USER_NAME -p $DB_NAME > $LOCATION/$FILE_NAME
echo "#Deleting Older than $BACKUP_DATE day(s) backup......"
sudo find $LOCATION -type f -mtime +$BACKUP_DATE -name '*.backup' -execdir rm -- '{}' \;
echo "==================================== End $NOW ===================================="

backup.exp: 

#!/usr/bin/expect set homeDirectory "/home/user/DB_BACKUP"
set dbPassword "db123" # Start logging
log_file $homeDirectory/LOG/db_backup.log; # Execute database backup script
spawn $homeDirectory/./backupexecutor.sh # Sending password to execute this command
expect "*asswor*"
send -- "$dbPassword\r" expect eof

Now, make those files executable for all:

$ chmod a+x backupexecutor.sh
$ chmod a+x backup.exp

Now, we need to set up a scheduler in Linux to execute backup.exp.  We are using crontab to schedule database backup. crontab has five fields to schedule a script/command

{minute} {hour} {day of month} {month} {day of week [0-6] Sunday = 0, Saturday = 7}

So, to execute every day at 8 PM, our crontab command will be:

0 20 * * *

Here, * means anything.

Now, open crontab in edit mode:

$ crontab -e

And write there:

0 20 * * * /file/path/to/backup.exp

Save the file.

Now, every day at 8 PM, backup.exp will execute!

Log file:

Image title

Database backup location:

Image title

And that’s it!

Original Link

Hands-On Look at ZFS With MySQL

This post is a hands-on look at ZFS with MySQL.

In my previous post, I highlighted the similarities between MySQL and ZFS. Before going any further, I’d like you to be able to play and experiment with ZFS. This post shows you how to configure ZFS with MySQL in a minimalistic way on either Ubuntu 16.04 or Centos 7.

Installation

In order to be able to use ZFS, you need some available storage space. For storage — since the goal here is just to have a hands-on experience — we’ll use a simple file as a storage device. Although simplistic, I have now been using a similar setup on my laptop for nearly three years (just can’t get rid of it; it is too useful). For simplicity, I suggest you use a small Centos7 or Ubuntu 16.04 VM with one core, 8GB of disk and 1GB of RAM.

First, you need to install ZFS as it is not installed by default. On Ubuntu 16.04, you simply need to run:

root@Ubuntu1604:~# apt-get install zfs-dkms zfsutils-linux

On RedHat or Centos 7.4, the procedure is a bit more complex. First, we need to install the EPEL ZFS repository:

[root@Centos7 ~]# yum install http://download.zfsonlinux.org/epel/zfs-release.el7_4.noarch.rpm
[root@Centos7 ~]# gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[root@Centos7 ~]# gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Apparently, there were issues with ZFS kmod kernel modules on RedHat/Centos. I never had any issues with Ubuntu (and who knows how often the kernel is updated?). Anyway, it is recommended that you enable kABI-tracking kmods. Edit the file /etc/yum.repos.d/zfs.repo, disable the ZFS repo, and enable the zfs-kmod repo. The beginning of the file should look like:

[zfs]
name=ZFS on Linux for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7.4/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.4/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
...

Now, we can proceed and install ZFS:

[root@Centos7 ~]# yum install zfs

After the installation, I have ZFS version 0.6.5.6 on Ubuntu and version 0.7.3.0 on Centos7. The version difference doesn’t matter for what will follow.

Setup

So, we need a container for the data. You can use any of the following options for storage:

  • A free disk device
  • A free partition
  • An empty LVM logical volume
  • A file

The easiest solution is to use a file, and so that’s what I’ll use here. A file is not the fastest and most efficient storage, but it is fine for our hands-on. In production, please use real devices. A more realistic server configuration will be discussed in a future post. The following steps are identical on Ubuntu and Centos. The first step is to create the storage file. I’ll use a file of 1~GB in /mnt. Adjust the size and path to whatever suits the resources you have:

[root@Centos7 ~]# dd if=/dev/zero of=/mnt/zfs.img bs=1024 count=1048576

The result is a 1GB file in /mnt:

[root@Centos7 ~]# ls -lh /mnt
total 1,0G
-rw-r--r--. 1 root root 1,0G 16 nov 16:50 zfs.img

Now, we will create our ZFS pool, mysqldata, using the file we just created:

[root@Centos7 ~]# modprobe zfs
[root@Centos7 ~]# zpool create mysqldata /mnt/zfs.img
[root@Centos7 ~]# zpool status pool: mysqldata state: ONLINE scan: none requested
config: NAME STATE READ WRITE CKSUM mysqldata ONLINE 0 0 0 /mnt/zfs.img ONLINE 0 0 0
errors: No known data errors
[root@Centos7 ~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mysqldata 79,5K 880M 24K /mysqldata

If you have a result similar to the above, congratulations, you have a ZFS pool. If you put files in /mysqldata, they are in ZFS.

MySQL Installation

Now, let’s install MySQL and play around a bit. We’ll begin by installing the Percona repository:

root@Ubuntu1604:~# cd /tmp
root@Ubuntu1604:/tmp# wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb
root@Ubuntu1604:/tmp# dpkg -i percona-release_*.deb
root@Ubuntu1604:/tmp# apt-get update
[root@Centos7 ~]# yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm

Next, we install Percona Server for MySQL 5.7:

root@Ubuntu1604:~# apt-get install percona-server-server-5.7
root@Ubuntu1604:~# systemctl start mysql
[root@Centos7 ~]# yum install Percona-Server-server-57
[root@Centos7 ~]# systemctl start mysql

The installation command pulls all the dependencies and sets up the MySQL root password. On Ubuntu, the install script asks for the password, but on Centos7 a random password is set. To retrieve the random password:

[root@Centos7 ~]# grep password /var/log/mysqld.log
2017-11-21T18:37:52.435067Z 1 [Note] A temporary password is generated for root@localhost: XayhVloV+9g+

The following step is to reset the root password:

[root@Centos7 ~]# mysql -p -e "ALTER USER 'root'@'localhost' IDENTIFIED BY 'Mysql57OnZfs_';"
Enter password:

Since 5.7.15, the password validation plugin by defaults requires a length greater than 8, mixed cases, at least one digit and at least one special character. On either Linux distributions, I suggest you set the credentials in the /root/.my.cnf file like this:

[# cat /root/.my.cnf
[client]
user=root
password=Mysql57OnZfs_

MySQL Configuration for ZFS

Now that we have both ZFS and MySQL, we need some configuration to make them play together. From here, the steps are the same on Ubuntu and Centos. First, we stop MySQL:

# systemctl stop mysql

Then, we’ll configure ZFS. We will create three ZFS filesystems in our pool:

  • mysql will be the top-level filesystem for the MySQL-related data. This filesystem will not directly have data in it, but data will be stored in the other filesystems that we create. The utility of the mysql filesystem will become obvious when we talk about snapshots. Something to keep in mind for the next steps, the properties of a filesystem are by default inherited from the upper level.
  • mysql/data will be the actual datadir. The files in the datadir are mostly accessed through random IO operations, so we’ll set the ZFS record size to match the InnoDB page size.
  • mysql/log will be where the log files will be stored. By log files, I primarily mean the InnoDB log files. But the binary log file, the slow query log, and the error log will all be stored in that directory. The log files are accessed through sequential IO operations. We’ll thus use a bigger ZFS record size in order to maximize the compression efficiency.

Let’s begin with the top-level MySQL container. I could have used directly mysqldata, but that would somewhat limit us. The following steps create the filesystem and set some properties:

# zfs create mysqldata/mysql
# zfs set compression=gzip mysqldata/mysql
# zfs set recordsize=128k mysqldata/mysql
# zfs set atime=off mysqldata/mysql

I just set compression to gzip (the equivalent of GZIP level 6),  recordsize to 128KB, and  atime (the file’s access time) to off. Once we are done with the mysql filesystem, we can proceed with the data and log filesystems:

# zfs create mysqldata/mysql/log
# zfs create mysqldata/mysql/data
# zfs set recordsize=16k mysqldata/mysql/data
# zfs set primarycache=metadata mysqldata/mysql/data
# zfs get compression,recordsize,atime mysqldata/mysql/data
NAME PROPERTY VALUE SOURCE
mysqldata/mysql/data compression gzip inherited from mysqldata/mysql
mysqldata/mysql/data recordsize 16K local
mysqldata/mysql/data atime off inherited from mysqldata/mysql

Of course, there are other properties that could be set, but let’s keep things simple. Now that the filesystems are ready, let’s move the files to ZFS (make sure you stopped MySQL):

# mv /var/lib/mysql/ib_logfile* /mysqldata/mysql/log/
# mv /var/lib/mysql/* /mysqldata/mysql/data/

And then set the real mount points:

# zfs set mountpoint=/var/lib/mysql mysqldata/mysql/data
# zfs set mountpoint=/var/lib/mysql-log mysqldata/mysql/log
# chown mysql.mysql /var/lib/mysql /var/lib/mysql-log

Now we have:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mysqldata 1,66M 878M 25,5K /mysqldata
mysqldata/mysql 1,54M 878M 25K /mysqldata/mysql
mysqldata/mysql/data 890K 878M 890K /var/lib/mysql
mysqldata/mysql/log 662K 878M 662K /var/lib/mysql-log

We must adjust the MySQL configuration accordingly. Here’s what I put in my /etc/my.cnf file (/etc/mysql/my.cnf on Ubuntu):

[mysqld]
datadir=/var/lib/mysql
innodb_log_group_home_dir = /var/lib/mysql-log
innodb_doublewrite = 0
innodb_checksum_algorithm = none
slow_query_log = /var/lib/mysql-log/slow.log
log-error = /var/lib/mysql-log/error.log
server_id = 12345
log_bin = /var/lib/mysql-log/binlog
relay_log=/var/lib/mysql-log/relay-bin
expire_logs_days=7
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
pid-file=/var/run/mysqld/mysqld.pid

On Centos 7, selinux prevented MySQL from accessing files in /var/lib/mysql-log. I had to perform the following steps:

[root@Centos7 ~]# yum install policycoreutils-python
[root@Centos7 ~]# semanage fcontext -a -t mysqld_db_t "/var/lib/mysql-log(/.*)?"
[root@Centos7 ~]# chcon -Rv --type=mysqld_db_t /var/lib/mysql-log/

I could have just disabled selinux since it is a test server, but if I don’t get my hands dirty with selinux once in a while and with semanage and chcon, I will not remember how to do it. Selinux is an important security tool in Linux (but that’s another story).

At this point, feel free to start using your test MySQL database on ZFS.

Monitoring ZFS

To monitor ZFS, you can use the zpool command like this:

[root@Centos7 ~]# zpool iostat 3 capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
mysqldata 19,6M 988M 0 0 0 290
mysqldata 19,3M 989M 0 44 0 1,66M
mysqldata 23,4M 985M 0 49 0 1,33M
mysqldata 23,4M 985M 0 40 0 694K
mysqldata 26,7M 981M 0 39 0 561K
mysqldata 26,7M 981M 0 37 0 776K
mysqldata 23,8M 984M 0 43 0 634K

This shows the ZFS activity while I was loading some data. Also, the following command gives you an estimate of the compression ratio:

[root@Centos7 ~]# zfs get compressratio,used,logicalused mysqldata/mysql
NAME PROPERTY VALUE SOURCE
mysqldata/mysql compressratio 4.10x -
mysqldata/mysql used 116M -
mysqldata/mysql logicalused 469M -
[root@Centos7 ~]# zfs get compressratio,used,logicalused mysqldata/mysql/data
NAME PROPERTY VALUE SOURCE
mysqldata/mysql/data compressratio 4.03x -
mysqldata/mysql/data used 67,9M -
mysqldata/mysql/data logicalused 268M -
[root@Centos7 ~]# zfs get compressratio,used,logicalused mysqldata/mysql/log
NAME PROPERTY VALUE SOURCE
mysqldata/mysql/log compressratio 4.21x -
mysqldata/mysql/log used 47,8M -
mysqldata/mysql/log logicalused 201M -

In my case, the dataset compresses very well (4x). Another way to see how files are compressed is to use ls and du. ls returns the actual uncompressed size of the file, while du returns the compressed size. Here’s an example:

[root@Centos7 mysql]# -lah ibdata1
-rw-rw---- 1 mysql mysql 90M nov 24 16:09 ibdata1
[root@Centos7 mysql]# du -hs ibdata1
14M ibdata1

I really invite you to further experiment and get a feeling of how ZFS and MySQL behave together.

Snapshots and Backups

A great feature of ZFS that work really well with MySQL is snapshots. A snapshot is a consistent view of the filesystem at a given point in time. Normally, it is best to perform a snapshot while a flush tables with read lock is held. That allows you to record the master position, and also to flush MyISAM tables. It is quite easy to do that. Here’s how I create a snapshot with MySQL:

[root@Centos7 ~]# mysql -e 'flush tables with read lock;show master status;\! zfs snapshot -r mysqldata/mysql@my_first_snapshot'
+---------------+-----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+-----------+--------------+------------------+-------------------+
| binlog.000002 | 110295083 | | | |
+---------------+-----------+--------------+------------------+-------------------+
[root@Centos7 ~]# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
mysqldata/mysql@my_first_snapshot 0B - 24K -
mysqldata/mysql/data@my_first_snapshot 0B - 67,9M -
mysqldata/mysql/log@my_first_snapshot 0B - 47,8M -

The command took about one second. The only time where such commands would take more time is when there are MyISAM tables with a lot of pending updates to the indices, or when there are long running transactions. You probably wonder why the USED column reports 0B. That’s simply because there were no changes to the filesystem since the snapshot was created. It is a measure of the amount of data that hasn’t been free because the snapshot requires the data. Said otherwise, it how far the snapshot has diverged from its parent. You can access the snapshot through a clone or through ZFS as a file system. To access the snapshot through ZFS, you have to set the snapdir parameter to “visible, ” and then you can see the files. Here’s how:

[root@Centos7 ~]# zfs set snapdir=visible mysqldata/mysql/data
[root@Centos7 ~]# zfs set snapdir=visible mysqldata/mysql/log
[root@Centos7 ~]# ls /var/lib/mysql-log/.zfs/snapshot/my_first_snapshot/
binlog.000001 binlog.000002 binlog.index error.log ib_logfile0 ib_logfile1

The files in the snapshot directory are read-only. If you want to be able to write to the files, you first need to clone the snapshots:

[root@Centos7 ~]# zfs create mysqldata/mysqlslave
[root@Centos7 ~]# zfs clone mysqldata/mysql/data@my_first_snapshot mysqldata/mysqlslave/data
[root@Centos7 ~]# zfs clone mysqldata/mysql/log@my_first_snapshot mysqldata/mysqlslave/log
[root@Centos7 ~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mysqldata 116M 764M 26K /mysqldata
mysqldata/mysql 116M 764M 24K /mysqldata/mysql
mysqldata/mysql/data 67,9M 764M 67,9M /var/lib/mysql
mysqldata/mysql/log 47,8M 764M 47,8M /var/lib/mysql-log
mysqldata/mysqlslave 28K 764M 26K /mysqldata/mysqlslave
mysqldata/mysqlslave/data 1K 764M 67,9M /mysqldata/mysqlslave/data
mysqldata/mysqlslave/log 1K 764M 47,8M /mysqldata/mysqlslave/log

At this point, it is up to you to use the clones to spin up a local slave. Like for the snapshots, the clone only grows in size when actual data is written to it. ZFS records that haven’t changed since the snapshot was taken are shared. That’s huge space savings. For a customer, I once wrote a script to automatically create five MySQL slaves for their developers. The developers would do tests, and often replication broke. Rerunning the script would recreate fresh slaves in a matter of a few minutes. My ZFS snapshot script and the script I wrote to create the clone based slaves are available here.

Optional Features

In the previous post, I talked about a SLOG device for the ZIL and the L2ARC, a disk extension of the ARC cache. If you promise to never use the following trick in production, here’s how to speed MySQL on ZFS drastically:

[root@Centos7 ~]# dd if=/dev/zero of=/dev/shm/zil_slog.img bs=1024 count=131072
131072+0 enregistrements lus
131072+0 enregistrements écrits
134217728 octets (134 MB) copiés, 0,373809 s, 359 MB/s
[root@Centos7 ~]# zpool add mysqldata log /dev/shm/zil_slog.img
[root@Centos7 ~]# zpool status pool: mysqldata state: ONLINE scan: none requested
config: NAME STATE READ WRITE CKSUM mysqldata ONLINE 0 0 0 /mnt/zfs.img ONLINE 0 0 0 logs /dev/shm/zil_slog.img ONLINE 0 0 0
errors: No known data errors

The data in the SLOG is critical for ZFS recovery. I performed some tests with virtual machines, and if you crash the server and lose the SLOG, you may lose all the data stored in the ZFS pool. Normally, the SLOG is on a mirror in order to lower the risk of losing it. The SLOG can be added and removed online.

I know I asked you to promise to never use an SHM file as SLOG in production. Actually, there are exceptions. I would not hesitate to temporarily use such a trick to speed up a lagging slave. Another situation where such a trick could be used is with Percona XtraDB Cluster. With a cluster, there are multiple copies of the dataset. Even if one node crashed and lost its ZFS filesystems, it could easily be reconfigured and reprovisioned from the surviving nodes.

The other optional feature I want to cover is a cache device. The cache device is what is used for the L2ARC. The content of the L2ARC is compressed as the original data is compressed. To add a cache device (again, an SHM file), do:

[root@Centos7 ~]# dd if=/dev/zero of=/dev/shm/l2arc.img bs=1024 count=131072
131072+0 enregistrements lus
131072+0 enregistrements écrits
134217728 octets (134 MB) copiés, 0,272323 s, 493 MB/s
[root@Centos7 ~]# zpool add mysqldata cache /dev/shm/l2arc.img
[root@Centos7 ~]# zpool status pool: mysqldata state: ONLINE scan: none requested
config: NAME STATE READ WRITE CKSUM mysqldata ONLINE 0 0 0 /mnt/zfs.img ONLINE 0 0 0 logs /dev/shm/zil_slog.img ONLINE 0 0 0 cache /dev/shm/l2arc.img ONLINE 0 0 0
errors: No known data errors

To monitor the L2ARC (and also the ARC), look at the file: /proc/spl/kstat/zfs/arcstats. As the ZFS filesystems are configured right now, very little will go to the L2ARC. This can be frustrating. The reason is that the L2ARC is filled by the elements evicted from the ARC. If you recall, we have set primarycache=metatdata for the filesystem containing the actual data. Hence, in order to get some data to our L2ARC, I suggest the following steps:

[root@Centos7 ~]# zfs set primarycache=all mysqldata/mysql/data
[root@Centos7 ~]# echo 67108864 > /sys/module/zfs/parameters/zfs_arc_max
[root@Centos7 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@Centos7 ~]# grep '^size' /proc/spl/kstat/zfs/arcstats
size 4 65097584

It takes the echo command to drop_caches to force a re-initialization of the ARC. Now, InnoDB data starts to be cached in the L2ARC. The way data is sent to the L2ARC has many tunables, which I won’t discuss here. I chose 64MB for the ARC size mainly because I am using a low-memory VM. A size of 64MB is aggressively small and will slow down ZFS if the metadata doesn’t fit in the ARC. Normally, you should use a larger value. The actual good size depends on many parameters like the filesystem system size, the number of files and the presence of an L2ARC. You can monitor the ARC and L2ARC using the arcstat tool that comes with ZFS on Linux (when you use Centos 7). With Ubuntu, download the tool from here.

Removal

So the ZFS party is over? We need to clean up the mess! Let’s begin:

[root@Centos7 ~]# systemctl stop mysql
[root@Centos7 ~]# zpool remove /dev/shm/l2arc.img
[root@Centos7 ~]# zpool remove mysqldata /dev/shm/zil_slog.img
[root@Centos7 ~]# rm -f /dev/shm/*.img
[root@Centos7 ~]# zpool destroy mysqldata
[root@Centos7 ~]# rm -f /mnt/zfs.img
[root@Centos7 ~]# yum erase spl kmod-spl libzpool2 libzfs2 kmod-zfs zfs

The last step is different on Ubuntu:

root@Ubuntu1604:~# apt-get remove spl-dkms zfs-dkms libzpool2linux libzfs2linux spl zfsutils-linux zfs-zed

Conclusion

With this guide, I hope I provided a positive first experience in using ZFS with MySQL. The configuration is simple, and not optimized for performance. However, we’ll look at more realistic configurations in future posts.

Original Link

Orchestrating and Monitoring Hybrid Windows-Linux Container Environments

In our previous blog post on Microsoft Windows container monitoring, we talked about the new developments around Windows containers and how they are different from what we have seen in the Linux space. In this second part, we continue this discussion by also considering orchestration technologies and their impact on Windows containers, including hybrid container environments.

Orchestrator Choices for Windows and Hybrid Clusters

Orchestrators like Kubernetes, Docker Swarm, and Docker EE are a vital component when moving containers into production. Orchestrators manage your nodes and container workloads with the goal of maintaining a stable and scalable environment. This includes auto-scaling and self-healing capabilities, e.g. taking nodes offline for abnormal behavior, restarting containers, and the possibility of setting resource constraints to maximize your cluster usage.

Since Microsoft released Docker support on Windows, we witnessed some of the orchestrator technologies also adding in support for Windows containers and hybrid environments. Hybrid container environments are orchestration clusters where you have a mix of Linux and Windows hosts.

Bringing Microsoft Windows and Linux together into the same cluster environment brings some distinct advantages to your environment. As Microsoft is moving into the Linux space with the release of Microsoft SQL Server for Linux and dotnet/core, it is becoming easier as a Microsoft developer to use the power of Linux servers in your environment.

Enterprises usually also still have a set of legacy applications that need to run on Windows. In this case, a hybrid environment makes it easier to let the new and old work together, as it allows you to pick the platform that matches your problem set while still keeping everything in the same familiar Docker environment.

Swarm vs. Kubernetes

Docker Swarm and Docker EE were one of the first to support hybrid clouds in a container-native way. Docker Swarm is aware what type of container you are running and will automatically orchestrate on the right OS type.

Image title

Kubernetes added alpha support for Windows hosts in Kubernetes 1.5, improving it in later releases. At the moment, you need to manually set the nodeSelector to the Windows nodes for Windows applications.

Image title

Security

The big advantage of running a hybrid environment is that you could run some containers in a high-security mode using the Microsoft Windows Hyper-V container mode. These containers exist in a completely isolated environment, but with configuration, they can still talk to any of the other containers in your cluster. This gives you the benefits of containers, but without the overhead of running a full-fledged operating system.

This means that you could, for example, use Windows Hyper-V containers as gateways for your Linux containers.

Image title

Container Networking

The teams at Microsoft have been putting in a lot of effort to improve the Network stack for Kubernetes/Swarm support. The work that Microsoft has put in means that your Linux and Windows containers can share the same network space, allowing containers from one operating system to easily communicate with the other without the need to set up a complicated network environment. The Windows side does require some setup, but that is a fairly straightforward process.

Image title

What Does it Mean for Monitoring?

Running hybrid Windows-Linux containers is great, but now you also need to make sure that you can obtain visibility in such environments. This is important to ensure that your containers are running as expected and to quickly detect any performance issues with your services that are spread out over a hybrid environment.

However, many tools for container monitoring only support Linux containers, leaving the Windows part as a blind spot. Obviously, this is not desired, and having to use different tools to monitor different parts of a hybrid environment is also not an ideal situation. Even if you are not running Windows containers right now, you might do so in the future, and it is best to choose a monitoring tool that is flexible enough to deal with different environments.

Image title

Azure Container Service

Besides running your own Kubernetes cluster, you can also opt for a hosted Kubernetes solution. For Windows users, the Azure cloud and the new Azure Container Service (AKS) would the logical choice. 

Original Link

Helping Friends on the Linux Command-Line

I noticed that when I was helping a friend on the Linux command-line I was struggling. Spelling the commands over the phone or trying to read their screen over a bad Skype or Hangout screen sharing connection is not really fun. That’s why I spent the time to create something that works (for me at least). It allows your friend to connect to your Linux server on which you can open a shell on your friend’s computer.

Security

The connection is secured as it listens only on the localhost of your friend’s computer and the server. It is a bash shell that is tunneled over SSH to the server using socat. With the tmux terminal program (a younger brother of GNU screen) you can make a terminal that multiple people can view and interact with. This way you can let people show you things while you are on the phone and you can immediately correct it when it is going wrong. Also, the other party can see what you do and learn from it. Or if you are doing something that they don’t like then they can terminate the session.

Script-fu

These are the main commands of the script:

ssh -R 6000:localhost:6000 -N user@hostname &
tmux new-session -d -s support
socat exec:"tmux attach -t support",pty,raw,echo=0,stderr,setsid,sigint \ tcp-listen:6000,bind=localhost,reuseaddr &
tmux attach -t support

These above commands you should run on the client, while the following should run on the server:

socat file:`tty`,raw,echo=0 tcp-connect:localhost:6000

You should adjust the “user@hostname” to the rendezvous server that you are using.

Workflow

When your friend calls you just let him or her connect to your server with socat.sh from my cli-support repository. They most probably already have a web hosting account on your server and if not you spin up a VM for the occasion. Next, you connect to the server and issue the appropriate server command. From that moment you can see the shell of your friend and you can both type. It is recommended that you also set up an audio connection so that you can talk to each other explaining what you are doing (or trying to do).

Conclusion

I created a tool that helped me helping a few friends with simple Linux problems. It works for me and I can see what they are doing wrong and they can see me do it in a different way and learn from it. These problems have been programming, Git, SSH, and DNS related and in most cases I was able to help them out with this tool. There are two things I still want to improve: adjustable screen size on the server (it defaults to 80×25) and a simple delivery of the tool (I now email the installation commands). Other than that I have not found any issues yet. Does it work for you?

Click here to download the cli-support software from Github.

Original Link

MySQL and Linux Context Switches

In this blog post, I’ll look at MySQL and Linux context switches and what is the normal number per second for a database environment.

You might have heard many times about the importance of looking at the number of context switches to indicate if MySQL is suffering from the internal contention issues. I often get questions about what is a “normal” or “acceptable” number, and at what point you should worry about the number of context switches per second.

First, let’s talk about what context switches are in Linux. This StackOverflow Thread provides a good discussion, with a lot of details, but basically, it works like this.

The process (or thread, in MySQL’s case) is running its computations. Sooner or later, it has to do some blocking operation: disk IO, network IO, block waiting on a mutex or yield. The execution switches to the other process, and this is called voluntary context switch. On the other hand, the process/thread may need to be preempted by the scheduler because it used an allotted amount of CPU time (and now other tasks need to run) or because it is required to run high priority task. This is called involuntary context switches. When all the process in the system are added together and totaled, this is the system-wide number of context switches reported (using, for example, vmstat):

root@nuc2:~# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
17 0 0 12935036 326152 2387388 0 0 0 5 0 1 9 0 91 0 0
20 0 0 12933936 326152 2387384 0 0 0 3 32228 124791 77 22 1 0 0
17 0 0 12933348 326152 2387364 0 0 0 11 33212 124575 78 22 1 0 0
16 0 0 12933380 326152 2387364 0 0 0 78 32470 126100 78 22 1 0 0

This is a global number. In many cases, however, it is better to look at it as context switches per CPU logical core. This is because cores execute tasks independently. As such, they have mostly independent causes for context switches. If you have a large number of cores, there can be quite a difference:

The number of context switches per second on this system looks high (at more than 1,000,000). Considering it has 56 logical cores, however, it is only about 30,000 per second per logical core (which is not too bad).

So how do we judge if the number of context switches is too high in your system? One answer is that it is too high if you’re wasting too much CPU on context switches. This brings up the question: How many context switches can the system handle if it is only doing context switches?

It is easy to find this out!

Sysbench has a “threads” test designed specifically to measure this. For example:

sysbench --thread-locks=128 --time=7200 --threads=1024 threads run

Check the vmstat output or the Context Switches PMM graph:

We can see this system can handle up to 35 million context switches per second in total (or some 500K per logical CPU core on average).

I don’t recommend using more than 10% of CPU resources on context switching, so I would try to keep the number of the context switches at no more than 50K per logical CPU core.

Now let’s think about context switches from the other side: How many context switches do we expect to have at the very minimum for given load? Even if all the stars align and your query to MySQL doesn’t need any disk IO or context switches due to waiting for mutexes, you should expect at least two context switches: one to the client thread which processes the query and one for the query response sent to the client.

Using this logic, if we have 100,000 queries/second, we should expect 200,000 context switches at the very minimum.

In the real world, though, I would not worry about contention being a big issue if you have fewer than ten context switches per query.

It is worth noting that in MySQL not every contention results in a context switch. InnoDB implements its own mutexes and RW-locks, which often try to “spin” to wait for a resource to become available. This wastes CPU time directly rather than doing a context switch.

Summary

  • Look at the number of context switches per logical core rather than the total for easier-to-compare numbers.
  • Find out how many context switches your system can handle per second, and don’t get too concerned if your context switches are no more than 10% of that number.
  • Think about the number of context switches per query: the minimum possible is two, and values less than ten make contention an unlikely issue.
  • Not every MySQL contention results in a high number of context switches.

Original Link

Database App Development and Deployment to Linux in RAD Studio

The Devart company recently released UniDAC with support for Linux 64-bit platform. UniDAC makes application development and maintenance easier and simpler because the use of Direct mode in a custom application does not require the installation of client libraries, additional drivers, etc. This helps to avoid the overhead when accessing a DBMS, hence increasing performance.

So, in this article, we will demonstrate the capability of UniDAC to establish a connection to various DBMSs in Direct mode:

  • ASE
  • DBF
  • MySQL
  • Oracle
  • PostgreSQL
  • SQLite
  • SQL Azure
  • SQL Server

Creating a Console Application

Let’s create a new UniDAC application for Linux 64-bit in RAD Studio 10.2 Tokyo.

Go to File > New > Other:

Image title

In the dialog box that appears, click Console Application.

Image title

Configuring UniDAC to Connect in the Direct Mode

To use UniDAC in a console application, you should add the Uni unit in the uses section, as well as the unit with the UniDAC provider for each DBMS. Let’s add providers for all DBMSs which are mentioned at the beginning of the article:

program UniDAC_Linux;
{$APPTYPE CONSOLE}
{$R *.res}
uses SysUtils, Uni, ASEUniProvider, // add this unit for ASE DBFUniProvider, // add this unit for DBF MySQLUniProvider, // add this unit for MySQL OracleUniProvider, // add this unit for Oracle PostgreSQLUniProvider, // add this unit for PostgreSQL SQLiteUniProvider, // add this unit for SQLite SQLServerUniProvider; // add this unit for SQL Server & SQL Azure 

Creating connection and dataset instances, as well as executing SQL queries and data fetching are similar for all UniDAC providers:

var UniConnection: TUniConnection; UniQuery: TUniQuery;
begin UniConnection := TUniConnection.Create(nil); UniQuery := TUniQuery.Create(nil); UniQuery.Connection := UniConnection;

Establishing Connection and Data Fetching in Direct Mode

The Deployment tab contains only the application file; there are no additional libraries or files:

Image title

Delphi-code for ASE:

UniConnection.ProviderName := 'ASE';
UniConnection.SpecificOptions.Values['Direct'] := 'True';
UniConnection.Server := 'ASE_DB';
UniConnection.Port := 5000;
UniConnection.Username := 'sa';
UniConnection.Password := '*****';
UniConnection.Database := 'DEMO';
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC ASE Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select @@version');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

ASE execution result:

Image title

Delphi-code for DBF:

UniConnection.ProviderName := 'DBF';
UniConnection.SpecificOptions.Values['DBFFormat'] := 'dfdBaseVII';
UniConnection.SpecificOptions.Values['Direct'] := 'True';
UniConnection.Database := '/home/test/Documents';
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC DBF Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

DBF execution result:

Image title

Delphi-code for MySQL:

UniConnection.ProviderName := 'MySQL';
UniConnection.Server := 'MySQL_db';
UniConnection.Port := 3312;
UniConnection.Username := 'root';
UniConnection.Password := '*****';
UniConnection.Database := 'demo';
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC MySQL Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select @@version');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

MySQL execution result:

Image title

Delphi-code for Oracle:

UniConnection.ProviderName := 'Oracle';
UniConnection.SpecificOptions.Values['Direct'] := 'True';
UniConnection.Server := 'ORCL12C:1521/pdborcl';
UniConnection.Username := 'scott';
UniConnection.Password := '******';
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC Oracle Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select * from v$version');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

Oracle execution result:

Image title

Delphi-code for PostgreSQL:

UniConnection.ProviderName := 'PostgreSQL'; UniConnection.Server := 'pg_db'; UniConnection.Database := 'demo'; UniConnection.Username := 'postgres'; UniConnection.Password := '******'; UniConnection.Port := 5432; UniConnection.Connect; Writeln(#13#10+'--== UniDAC PostgreSQL Provider ==--'+#13#10); WritelnQuery(UniQuery, 'select version()');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

PostgreSQL execution result:

Image title

Delphi-code for SQLite:

UniConnection.ProviderName := 'SQLite';
UniConnection.SpecificOptions.Values['Direct'] := 'True';
UniConnection.SpecificOptions.Values['ForceCreateDatabase'] := 'True';
UniConnection.Database := ':memory:';
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC SQLite Provider ==--'+#13#10);
Writeln(UniConnection.ServerVersionFull);
SQLite_Create_EMP(UniQuery);
SQLite_Insert_EMP(UniQuery);
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

SQLite execution result:

Image title

Delphi-code for SQL Azure:

UniConnection.Disconnect;
UniConnection.ProviderName := 'SQL Server';
UniConnection.SpecificOptions.Values['Provider'] := 'prDirect';
UniConnection.Server := 'qps1hrvdke.database.windows.net';
UniConnection.Database := 'DEMO';
UniConnection.Username := '****@qps1hrvdke';
UniConnection.Password := '*******';
UniConnection.Port := 1433;
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC SQL Server(Azure) Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select @@version');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

SQL Azure execution result:

Image title

Delphi-code for SQL Server:

UniConnection.Disconnect;
UniConnection.ProviderName := 'SQL Server';
UniConnection.SpecificOptions.Values['Provider'] := 'prDirect';
UniConnection.Server := '192.168.0.15\MSSQL2016';
UniConnection.Database := 'DEMO';
UniConnection.Username := 'sa';
UniConnection.Password := '*****';
UniConnection.Port := 1433;
UniConnection.Connect;
Writeln(#13#10+'--== UniDAC SQL Server Provider ==--'+#13#10);
WritelnQuery(UniQuery, 'select @@version');
WritelnQuery(UniQuery, 'select empno, ename, job, hiredate from emp');

SQL Server execution result:

Image title

Here is the complete project source code.

Conclusion

In this article, we showed how easily and simply you can create applications for Linux without deploying additional files, client libraries, or drivers. For this, you only need to install UniDAC, write a few code lines, and the application to work with databases on Linux is complete.

Original Link

Replacing Modern Tools With Retro Linux Commands

Your boss walks in looking anxious but still composed, you look up and they ask, “Can you tell me if the (antiqued system you have never heard of) is having any issues and what sort of load it is under? (Team you have also never heard of) is complaining. While you are in (analytics tool) can you generate an Excel file so I can toy around?”

After some investigation you find an old busted server running an ancient version of Linux. You can’t connect it to your awesome clustered log analytics tool, you are in the weeds, you are alone, you are in trouble.

No worries, ancient Linux tools to the rescue!

WARNING: Some of these chained commands are absurd and probably should never be used in real life. Hopefully, they act as useful introductions to various Linux tools and the types of things you can do. If you end up with a mess of chained commands, consider a scripting language.

First, time to see where this esoteric system puts its log files. We use locate as our first pass.

locate .log

What is going on?: locate checks a database or databases maintained by updatedb. This database can be out of date or missing data so it isn’t as exhaustive as find and is just based on the data it has indexed.

For a more exhaustive search, we can reach out to find.

find / -name \*.log\*

What is going on?: find checks the file system in real-time, so it is much slower than locate, but will always be up to date. This search starts in / (the root of the file system) and searches all filenames (-name) that contain.log in any part of them.

Woah, way too many, but still not sure exactly what I am looking for, so I start filtering out stuff I absolutely don’t want.

locate .log | grep -v kern.log | grep -v other.log

What is going on?: We are piping the output of locate into grep and using the -v flag to remove lines that match that substring. We do this twice so the result intentionally lacks anything with kern.log or other.log. It could be done with a single grep command, but I leave that as an exercise for the reader.

Success, you found the esoteric system files under a random home directory. It must have its own user.

/export/home/random/random.log
/export/home/random/random.log.1.gz
/export/home/random/random.log.2.gz
/export/home/random/random.log.3.gz
/export/home/random/random.log.4.gz

Now you want to quickly look at the content of a complete log, luckily we have zcat which lets us get plain text out of a gzipped file.

Note: you will find a handful of utilities have the z-equivalents that work directly on gz files: zgrep, zdiff, and more.

So, first and foremost the system is acting ‘weird’. Time to check for recent errors. Use the power of grep!

zcat /export/home/random/random.log.2.gz | grep -i error

What is going on?: We are piping the output of the zcat into grep and using the -i flag to case-insensitively match lines that include the word error.

The output is many screens full of errors, so you get a count quickly using wc.

zcat /export/home/random/random.log.2.gz | grep -i error | wc -l

What is going on?: We are extending our last example with wc -l to count the number of lines (rather than words) in the output.

10033 … that is a lot of errors! Time to find out how many different types of errors we are getting. Also, time to investigate the error format by looking at a few of the errors at the start by using head which by default will return 10, but we only need 5. We could have also used tail to grab from the end.

zcat /export/home/random/random.log.2.gz | head -5

What is going on?: We are changing our last example to remove the line count and instead just return the first 5 lines using head.

Note: A common use for tail is monitoring files as they are changed using tail -f file_name.

The output of the above command is:

Apr 28 17:06:20 ip-172-31-11-241 random[12545]: reverse mapping checking getaddrinfo for 216-19-2-8.commspeed.net failed - POSSIBLE BREAK-IN ATTEMPT! [preauth]
Apr 28 17:06:21 ip-172-31-11-241 random[12545]: Received disconnect from 216.19.2.8: 11: Bye Bye [preauth]
Apr 29 17:06:22 ip-172-31-11-241 random[12547]: Invalid user admin from 216.19.2.8 [auth]
Apr 29 17:06:23 ip-172-31-11-241 random[12547]: input_userauth_request: invalid user admin [preauth]
Apr 29 17:06:24 ip-172-31-11-241 random[12547]: questionable user request: delete everything [usage]

Hmph, that usage one is interesting, let’s see how many of those there are. But we don’t want to use grep because it might find the wrong things, so we need to isolate the specific field we want, cut to the rescue.

Note: This could also have been done with either sed or awk.

zcat /export/home/random/random.log.2.gz | cut -d "[" -f 3 | cut -d "]" -f 1

What is going on?: We now have added in the cut command, we set up a delimiter to split the string on, the first thing we do is split our lines on the “[” char, which turns

 Apr 28 17:06:21 ip-172-31-11-241 random[12545]: Received disconnect from 216.19.2.8: 11: Bye Bye [preauth]

into

 Apr 28 17:06:21 ip-172-31-11-241 random 12545]: Received disconnect from 216.19.2.8: 11: Bye Bye preauth]

Then we take the 3rd field “preauth]” and we split it again using cut on “]” which leaves us with just:

 preauth

Note: There are a lot of tools we could have used to trim the last “]”, sed or awk would work here too.

Perfect, that (for our 5 line examples above) gives us the following output:

preauth
preauth
auth
preauth
usage

Terrific, now we want to get counts of each one so we can add in a bit of sort and uniq magic.

zcat /export/home/random/random.log.2.gz | cut -d "[" -f 3 | cut -d "]" -f 1 | sort | uniq -c | sort -bnr

What is going on?: We pipe the output of our last command to sort, giving us the output:

auth
preauth
preauth
preauth
usage

Then we use uniq to reduce to unique words and by adding -c we prefix each line with the count.

 1 auth 3 preauth 1 usage

We pipe it back into sort with -bnr which will (-b) ignore leading blanks, (-n) order using numeric value, and (-r) reverse the order putting highest values first.

 3 preauth 1 usage 1 auth

From this, we extract some interesting numbers and clues to the problem, but really need it also grouped by day to find patterns. So this one introduces a few new things. Here we actually use awk, see the first appearance of xargs, and zgrep.

zcat /export/home/random/random.log.2.gz | awk '{print $1" "$2}' | uniq | xargs -I '{}' sh -c ' echo "{}"; zgrep "{}" /export/home/random/random.log.2.gz | cut -d "[" -f 3 | cut -d "]" -f 1 | sort | uniq -c' | sort -bnr

What is going on?: We start by piping the compressed file into awk, which takes the first two fields and discards the rest. The first two fields are the month and day. Then we remove duplicate days using unique and are left with a series of days like “Apr 28” and “Apr 29.” At this point, we take those days and pipe them into xargs, which will run the given command for each line passed into it, in this case, each day. We output the day with echo, then we grep the file for that day and pipe matches into the same series of steps we came up with before

That outputs:

Apr 28 2 preauth
Apr 29 1 auth 1 preauth 1 usage ...
May 5 5152 usage 4 auth 2 preauth
...

Great, we found the troublesome day, time to investigate that date further… but before we do that we still owe our boss an Excel file, luckily he doesn’t really care whether it is an Excel file as long as it loads in Excel, so CSV to the rescue. Well, let’s just get ALL the data in CSV format and let them deal with it.

echo "date, time, ip, process, pid, message, type" > boss_output.csv && zcat /export/home/random/random.log.2.gz | sed -n 's/\(\w\+\) \(\w\+\) \(\w\+:\w\+:\w\+\) \(.[^ ]\+\) \(\w\+\)\[\(\w\+\)\]: \(.[^\[]*\) \[\(\w\+\).*/ , , , , , , /p' >> boss_output.csv

What is going on?: First, we create a new file by echoing a header line into boss_output.csv using redirection. After that, we pipe the uncompressed content of the log file into sed. Using a regular expression to break up the content we create a CSV file.

Now the boss output looks like the following which hopefully will let them make a pretty graph of some sort.

date, time, ip, process, pid, message, type
Apr 28, 17:06:20, ip-172-31-11-241, random, 12545, reverse mapping checking getaddrinfo for 216-19-2-8.commspeed.net failed - POSSIBLE BREAK-IN ATTEMPT!, preauth
Apr 28, 17:06:21, ip-172-31-11-241, random, 12545, Received disconnect from 216.19.2.8: 11: Bye Bye, preauth
Apr 29, 17:06:22, ip-172-31-11-241, random, 12547, Invalid user admin from 216.19.2.8, auth
Apr 29, 17:06:23, ip-172-31-11-241, random, 12547, input_userauth_request: invalid user admin, preauth
Apr 29, 17:06:24, ip-172-31-11-241, random, 12547, questionable user request: delete everything, usage
... (many more)

May the boss be satisfied.

Original Link

Inside the Open Mainframe Project

At the All Things Open conference held recently in Raleigh, NC, I sat down with John Mertic, director of program management for the Open Mainframe Project, as well as Leonard Santalucia, chair of the Open Mainframe Project Governing Board. I wanted to learn more about what the project is, what it does, and how it serves the developer community.

The Open Mainframe Project is a Linux Foundation Project — more mainframe organizations use the Linux OS than any other platform. “We’re driving the use and advocacy of Linux in the mainframe architecture,” Mertic explained. At its core, he said, the Open Mainframe Project intends to serve a focal point for deploying and using Linux in a mainframe environment. It’s also looking to bring the mainframe community together to share resources.

“Mainframe is not anything new. The oldest computing community is the world is mainframe. But mainframe is invisible,” Mertic said. “The Open Mainframe Project is all about making the mainframe visible to the world.”  

A large portion of the Open Mainframe Project’s goals touch upon education. The Techbriefs section of its website includes downloadable resources on various open source projects (Docker, MongoDB, and Blockchain) and their relationship to Linux on the mainframe. It includes a Community Forum where experts answer questions, and the project is on Github as well.

Its mission to educate also involves offering scholarships to students to attend conferences like All Things Open, partnering with schools and other organizations to get more students to see the mainframe as a viable career option, holding interviews with journalists like myself looking to understand the basics, generally breaking down barriers to open source adoption on the mainframe, and much more.

Original Link

Linux Tips and Tricks: From Picking a Distro to Using the Command Line

Linux is a vast ecosystem of operating systems. Unlike Windows or macOs variants, there are loads of Linux distributions (distros) available. But these distros often differ greatly. Whether you’re just getting started with Linux, or are a seasoned pro, here are the tips and tricks you need to know.

Picking the Right Linux Operating System

Whereas Windows and macOS offer fairly few choices for their operating system (OS) options, Linux presents a ton of flavors.

The Environment

First, determine what you plan to use your server for. If it’s a desktop or laptop environment, most Linux OSes with a graphical user interface (GUI) built in will fulfill your needs. 

However, for a server environment, you’ll want to consider a specialized Linux distro. Best picks for a Linux server OS include openSuse, CentOS, and Oracle Linux. Often, server operating systems include more tools than the average desktop OS. The difference in Ubuntu Desktop and Ubuntu Server illustrates this well. While Ubuntu Desktop includes a GUI, Server lacks this feature. Of course, you can install a desktop environment later on. Yet Ubuntu Server comes preloaded with server-specific programs and packages such as Apache2 and Bind9. During the installation process for Ubuntu Server, options include specialized flavors for web, email, samba, and file servers. 

Still, a desktop Linux OS can run a server. On my Xeon-powered Lenovo ThinkServer TS140, I’m using a Long Term Service (LTS) version of Ubuntu. Since my setup is primarily a media server and game server, I didn’t really need most of the features of Ubuntu Server. 

  • Desktop vs. server

  • GUI vs. no GUI

The Hardware

Aside from what you’re creating, whether a desktop, server, media center PC, retro gaming arcade, or something completely different, considering hardware is a must. Lightweight Linux distros are suited to older hardware, and as such feature desktop environments which use fewer system resources. What’s compatible on a laptop or desktop is drastically different than Linux OSes you can install on a Raspberry Pi. Therefore, think about what hardware you’re running as well as your environment. 

Ease of Use

The biggest misconception of Linux is that it’s difficult to use. It can be more challenging. But it doesn’t have to be. Opting to buy a pre-built system eschews installation. Even a do-it-yourself (DIY) install may be as simple as creating a live disk and installing to your environment of choice. 

For power users, modular installs are an option. Distros such as Gentoo or Nutyx offer Linux from scratch. Unlike the average desktop Linux distro, these OSes simply boot into the command line. Then, during the installtion process, the user slogs through everything from compiling a kernel to selecting a desktop environment. Most distros provide excellent documentation, so all it takes is a bit of patience. 

Switching From Windows or macOS

For those switching from Windows or macOS, try these tips for a smooth transition. 

Pick a User-Friendly OS

If you’re switching from Windows or macOS, pick a distribution that’s more familiar. Look for a GUI and a solid package manager. A few options are ChaltetOS, Manjaro, Pop_OS!, and Ubuntu. With distros like these, everything from initial install to general use is pretty intuitive. This allows you to gradually learn more complex Linux elements.

Best Linux distros for beginners:

  • ChatletOS

  • Manjaro

  • Pop_OS!
  • Ubuntu
  • Elementary OS

Software Compatibility

The biggest challenge in switching to Linux is software compatibility. If you’re ingrained in one ecosystem, you may find that your preferred apps and programs don’t work well.

Wine is one means to run Windows programs on Linux. It’s a compatibility layer which stands for Wine is Not an Emulator. I’ve used Wine to overcome the problem of getting my Windows programs running on Linux. Notably, this is an excellent means to run older apps. While my 64-Bit Windows 10 OS can’t load 16-Bit programs, I’m able to run 16-bit Windows apps on 64-bit Linux distros using Wine. Such as my beloved copy of Westwood’s 1997 Blade Runner PC game. Plus, various tools for Wine like the PlayOnLinux frontend make installing software in Wine even easier. There’s also the Vineyard project, a conglomeration of libraries and tools for managing Windows programs on Linux incredibly simple.

Alternatively, try searching for replacement programs. You can find tons of excellent notepad apps, open-source office software, and more. But you might be surprised to find that many of your favorite apps boast native Linux installers. It’s increasingly common that game developers release cross-platform titles, such as Alien: Isolation were graced with cross-platform compatibility

Learn the Command Line

While you may be able to avoid the command line for many tasks, it’s inevitable that you’ll need to use it. The command line is essential to Linux, and one of the main reasons to use a Linux distro. 

Common and Lesser-known Linux Commands You Should Know

Check out these common and lesser-known Linux commands you need to know.

Change Directory

cd

This command changes directory. For instance, you can navigate into subdirectories of your PC using this command. If I want to navigate into the Movies subfolder on my server, I simply enter:

cd /home/moe/Videos/Movies

However, some folders may feature spaces in their names. Attempting to change directory into these is possible, but you’ll need to throw quotes around these. 

cd /home/moe/Videos/"TV Shows"

You can also back up to a previous directory. So if we’ve entered: 

cd /home/moe/Videos/"TV Shows"

then to backtrack to the Videos folder, simply input:

cd -

Make Directory

Changing directories is really easy, as is making directories. To create a directory, use:

mkdir

Running 

mkdir stuff

creates a directory called stuff in the current directory. If using this method, you’ll need to change directory via the command line into the desired directory. You can also create a directory with a path:

mkdir -p /home/moe/x/y/z

What’s neat is that even if /home/moe/x/y is non-existent, this command will create that directory before spawning the y directory. 

Superuser Permissions

To execute some commands, you’ll need to use superuser permissions. Thus, superuser do is necessary:

sudo

For instance, installing software with the APT package manager and superuser permissions would look like:

sudo apt-get install [PACKAGE NAME]

If you forget to run a command with superuser permissions, you can add that in retrospectively with

sudo!!

I find this really useful for those times I forget to add the sudo command. 

apt-get install [PACKAGE NAME]
sudo!!

String Commands Together

A nifty feature is the functionality to string together commands in the command line. This is particularly handy when you want to perform updates and installations at the same time:

sudo apt-get update && sudo apt-get upgrade

Move Files via Command Line

If you’re managing a machine via the command line, you may need to move files. To perform such an action, use the command:

mv

For example: 

mv/home/moe/logfile /home/moe/Documents

The first portion shows the logfile, and the second part is where to move it. In this case, the Documents folder. So the file logfile will be moved from the directory moe to the Documents folder.

Add PPAs

PPAs are personal package archives. These are repositories of software not included in the default Linux OS install. To load certain software in Linux, it’s often required to add a PPA first.

sudo add-apt-repository [REPOSITORY NAME]

For a specific example, here’s how you would install Open Broadcaster Software for Linux:

sudo add-apt-repository ppa:obsproject/obs-studio
sudo apt update && sudo apt install obs-studio

First, this adds the required PPS for OBS. Then, it performs an update and installs the software by stringing commands together. 

List

The command 

ls

lists content in alphabetical order. This can be used in conjunction with loads of practical commands. Want to view a list in reverse alphabetical order for instance? Enter:

ls -r

Linux Tips and Tricks: Final Thoughts

While Linux may seem daunting for first-time users, it’s rather easy to use and there are loads of benefits. From the wealth of distro choices to open-source nature and fantastic, dedicated communities, there are loads of reasons to make the switch. It all starts with picking the right Linux OS for your needs. Before you install a distro, try booting from a live CD or running an operating system in a virtual machine first. This allows you to take OSes for a test run. Thankfully, you don’t even need to pick between operating systems. Instead, you can dual boot.

Your turn: What are your favorite Linux tips and tricks? Let us know in the comments!

Original Link

Saving H2O Model Objects as Text Locally

Sometimes, you may want to store an H2O model object as text to a local file system. In this example, I will show you how you can save H2O model objects to your local disk as simple text content. You can get the full-working Jupyter notebook for this example here from my GitHub.

Based on my experience, the following example works fine with Python 2.7.12 and Python 3.4. I also found that the H2O model object tables were not saved to text files from Jupyter notebook. However, when I ran the same code from the command line into the Python shell, all the content was written perfectly.

Let’s build an H2O GBM model using the public PROSTATE dataset. The following script is the full working script, which will generate the GBM binomial model:

import h2o
h2o.init() local_url = "https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/prostate.csv"
df = h2o.import_file(local_url) y = "CAPSULE"
feature_names = df.col_names
feature_names.remove(y) df[y] = df[y].asfactor() df_train, df_valid = df.split_frame(ratios=[0.9])
print(df_train.shape)
print(df_valid.shape) prostate_gbm = H2OGradientBoostingEstimator(model_id = "prostate_gbm", ntrees=1000, learn_rate=0.5, max_depth=20, stopping_tolerance=0.001, stopping_rounds=2, score_each_iteration=True) prostate_gbm.train(x = feature_names, y = y, training_frame=df_train, validation_frame=df_valid)
prostate_gbm

Save the model details to the disk as shown below:

old_target = sys.stdout
f = open('/Users/avkashchauhan/Downloads/model_output.txt', 'w')
sys.stdout = f

Let’s see the content of the local file we have just created in the above step (it is empty):

!cat /Users/avkashchauhan/Downloads/model_output.txt

Launch the following commands, which will fill the standard output buffer with the model details as text:

print("Model summary>>> model_object.show()")
prostate_gbm.show()

Push the standard output buffer to the text file, which is created locally:

sys.stdout = old_target

Check back with the local file contents. This time, you will see that the output of the above command is written into the file:

!cat /Users/avkashchauhan/Downloads/model_output.txt

You will see the command output stored into the local text file as below:

Model summary>>> model_object.show()
Model Details
=============
H2OGradientBoostingEstimator : Gradient Boosting Machine
Model Key: prostate_gbm ModelMetricsBinomial: gbm
** Reported on train data. ** MSE: 0.036289343297
RMSE: 0.190497620187
LogLoss: 0.170007804527
Mean Per-Class Error: 0.0160045361428
AUC: 0.998865964296
Gini: 0.997731928592
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.487417363665: Maximum Metrics: Maximum metrics at their respective thresholds Gains/Lift Table: Avg response rate: 40.36 % ModelMetricsBinomial: gbm
** Reported on validation data. ** MSE: 0.161786079676
RMSE: 0.402226403505
LogLoss: 0.483923658542
Mean Per-Class Error: 0.174208144796
AUC: 0.871040723982
Gini: 0.742081447964
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.205076283533: Maximum Metrics: Maximum metrics at their respective thresholds Gains/Lift Table: Avg response rate: 39.53 % Scoring History: Variable Importances:

Note: If you are wondering what the ! sign does here, it is used here to run a Linux shell command (in this case, cat  is the Linux command) inside the Jupyter cell.

That’s it; enjoy!

Original Link

Exploring DevOps: Work Smarter, Code Better

This month’s dive into DevOps is all about improving your code from the source- you! Learn techniques to work more efficiently, making your final product better (and your life easier). Read about principles of programming, tricks in Git, why you might want to switch to Linux, and more. Plus, you’ll find free ebook downloads and job opportunities. Happy coding!


5 Trending DevOps Articles on DZone

  1. SOLID Principles by Examples: Single Responsibility, by Michele Ferracin. This post kicks off a series about understanding the SOLID principles through simple examples to write high-cohesion, low-coupling code.

  2. Simple CRUD With Git, by Unni Mana. Learn to use the basic CRUD commands in Git, like creating a repo and adding or deleting files, to make your life easier and improve productivity.

  3. 8 Reasons to Learn and Switch to Linux, by Grzegorz Ziemoński. The benefits of switching to Linux span across multiple areas, including becoming a better professional, saving money, and having more fun. What are you waiting for?

  4. Configuration as Code With Docker and Spring Boot, by Ram Gopinathan. Learn how this DevOps practice makes life easier for operations, as well as delivering increased velocity to the software lifecycle.

  5. What I’m Talking About When I Talk About TDD, by Uberto Barbini. See how Test-Driven Design makes designing software architecture easier by allowing you to test as you go along and fix mistakes as they arise. 


DevOps Around the Web

  1. Keybase Git gets keys, basically: Secure chat app encrypts your repos, Thomas Claburn, October 5, 2017. This feature promises to ensure security, wherever you’re coding.

  2. Java SE 9 and Java EE 8 arrive, 364 days later than first planned, Simon Sharwood, September 22, 2017. Now that the long, long wait is over, let’s talk about the code. Check out a full list of the new features.

  3. Linux Foundation Aims to Advance Open-Source Software Development, Sean Michael Kerner, September 14, 2017. Check out this video to see why Linux and The Linux Foundation want to foster collaboration and open source development.


Dive Deeper Into DevOps

  1. DZone’s Guide to Automated Testing: Improving Application Speed and Quality: a free ebook download.

  2. Getting Started With Kubernetes: DZone’s updated Refcard on the open-source orchestration system for managing containerized applications across multiple hosts.


Who’s Hiring?

Here you can find a few opportunities from our Jobs community. See if any match your skills and apply online today!

DevOps Engineer
Qubole
Location: Santa Clara, CA, United States
Experience: Strong system administration background for Linux based systems. Experience working with config and deploy management tools like Chef or Puppet. Comfortable in scripting languages like Python, Ruby, and Bash.

DevOps Engineer
RISI
Location: Raleigh, NC, United States
Experience: Experience with Azure or AWS cloud services. Advanced Linux and Windows administration skills. Knowledge of container technologies including Docker, optionally Swarm or Kubernetes.

Original Link

8 Reasons to Learn and Switch to Linux

Over two months ago, I made a very important decision regarding my everyday workspace – I decided to switch from Windows to Linux as the main system on both my personal and work laptops. Aside from some minor driver issues in the beginning, the switch was smooth and I wouldn’t think about going back. In case you’ve been thinking about making such a move on your computers or, like me, you’ve been procrastinating on learning Linux for the past few years, I will try to give you some strong reasons to finally make the right decision.

1. Your Software Runs on Linux

Assuming that you’re a dev (which seems like a valid assumption to make about a DZone reader), there is a huge chance that the software you’re building runs on a daily basis on Linux machines. Want it or not, believe in the God of Abstraction or not, it has its implications and sooner or later you will have to deal with the system in one way or another. Wouldn’t it be better if you knew the system that runs your software before everything starts burning or you urgently need to configure something?

2. Many Tools Work Better/Only on Linux

Just to give you some examples, there’s been countless times when I’ve cursed on Docker support in Windows or hoped that, one day, tools like SDKMAN! will become available on Windows. Not to mention shells that “just work” on Linux without Cygwin magic and similar stuff. Tooling-wise, the switch to Linux has been a huge boost for me. There are probably some counterexamples, but so far I haven’t encountered any.

3. Linux Can Boost Your Productivity

Now, this is probably not an argument about capabilities of one system that the other does not have, but rather the immediate availability of certain things. Once you embrace the power of controlling your system almost exclusively using the keyboard, tools like Vim or Emacs for rapid text editing, and optimize your environment with the help of some apps and a good window manager, you’ll become a speed monster like you’ve never dreamed about. I’ll be honest here, I’m far from being such a speed monster myself, but I follow the stoics here:

“[..] if it is humanly possible, consider it to be within your reach”
– Marcus Aurelius

4. Learning Linux Will Expand Your Horizons

Ha, that’s a cliche, isn’t it? Learning ANYTHING will expand your horizons. Well, that’s true and Linux is a particularly good example of this. If you were to learn “ANYTHING” right now to improve your software engineering skills, why shouldn’t it be the operating system that the world runs on? Also, as much as I believe that most developers should learn to speak high-level, human, talk-to-me-like-a-5-year-old language, I also believe that they should get their hands dirty with the low-level stuff at one point. Therein lies the source of better understanding, inspiration, and gratefulness for those who coded decades before us.

5. Linux is Extremely Customizable

For almost 20 past years, the majority of the personal computers that I’ve seen have been running Windows. Since my very first one, running Windows 95 on some few hundred MHz processor, they’ve been all looking and working basically the same. A start menu, icons on the desktop, a bar with the application’s name and three buttons on top of every window. Of course, you can change a lot of that, but that’s not the way you work with a Windows computer. When talking about some lower-level stuff, the situation is pretty much the same. You install the system, probably some drivers and that’s about that. You can install any apps you want, but, for your own good, leave the system alone as it is!

Now, with Linux, it’s a totally different story. Depending on the distro you choose and your level of proficiency you can customize almost everything, from the system’s kernel, through compiling and installing apps, to every detail in the looks of the open windows. You can prepare your very own, unique version of the system, optimized specifically for your needs. Your own penguin that loves you above anything else in the world!

6. Linux Is Fun

Enough talking to your reasoned choice (at least for this paragraph) – wake up your inner nerd! Windows is BOOORING! Any system that does everything for you and gives you minimum flexibility is boring! I mean, it’s good for your grandma and maybe your parents but not for a programmer. Unless I’m too stereotypical here, you got into coding because you love playing and tinkering with computers, not because they pay you a lot or any other reason. Your very own operating system is a great place to play, tinker and optimize – utilize that!

7. Linux Is Free

To be honest, I didn’t know whether this argument should be one of the first or one of the last. It ended up here, but I still think it’s an important one. I mean, if you’re not an Apple or Microsoft employee, why the hell would you spend a lot of money for a piece of proprietary software that is not necessarily any better than what you can get for free? Unless you have a really good reason for any of the paid systems, they are simply a waste of money.

There’s also the other side of the word “free”. Linux is a truly free technology, in the sense that its community and many of the distributions work hard to make the best software available to people free of charge. As much as I love being paid to do programming, I believe that’s a noble goal worth noting here.

8. Linux Is Fast

This argument has ended up being the last one because it’s both relative, e.g. to how much many you can spend on a computer, and has a caveat (IME some distros like Ubuntu are not very lightweight). The point is that if you’re a fairly skilled Linux user, you do not need as much computer resources to have a blazing fast operating system as with Windows (and probably macOS as well). That’s beneficial to both the people who already own a computer (they get a speed boost) and people who are about to buy a computer (they can buy a lower spec one for less money).

Summary

As you can see, the benefits you get span across multiple areas, including being a better professional, saving some money, and having more fun. And these are just some arguments, the list is obviously far from complete. Anyway, the conclusion for me is that Linux is a great system, worth learning and using on a daily basis, especially by software developers.

What operating system are you using? Do you have any strong arguments in favor of any of the major operating systems? Looking forward to see your opinions in the comments!

Original Link

Sidechains and Lightning Networks with Rusty Russell

A sidechain is a blockchain you can move bitcoins into and back from, which may have different abilities than the bitcoin blockchain.

The lightning network is a caching layer for bitcoin, which allows instant sending and failover to the normal bitcoin network if something goes wrong.

Rusty Russell is an Australian free software programmer and advocate, known for his work on the Linux kernel’s networking subsytem and the Filesystem Hierarchy Standard.

Questions

  • What is a sidechain?
  • How do you define a blockchain?
  • What is a micropayment channel?
  • What is a lightning network?
  • How do lightning networks enable micropayments?
  • How do lightning networks enable purchase of a cup of coffee?
  • How do sidechains affect altcoins?

Links

Original Link