Beware, Here Be Plasma Ninjas

The Green HPCG List

2018-11-18T01:17:53+01:00

SC18 is in the books now and the latest TOP500 results are out. Perfect time for some play with numbers!

This post will focus one efficiency, in terms of performance per Watt, simply because system power envelope is a major constrain for upcoming Exascale systems. With the great numbers from TOP500, we try to extend theoretical estimates from theoretical Flop/Ws of individual compute hardware to system scale.

Of course, we do have the Green 500, but let us compare something different this time…

HPCG

Since recently, an additional benchmark complementing raw LINPACK performance (TOP500 metric) on systems was introduced called HPCG. The High Performance Conjugate Gradients benchmark project computes a conjugate gradient (CG) method which is contrary to the dense matrix operations in LINPACK a sparse task.

Why would this be interesting? Besides the official list of reasons on the project page and the use of the method itself for e.g. solving long-range potentials, CG is a good measure for many kinds of finite stencil-based codes such as fluid solvers, FDTD solvers, finite element solvers and the like.

CG is, contrary to LINPACK, quickly memory bound and reasonably communication intensive at scale, just like most real-world science problems. That means for a system benchmark, well-designed networking and per-node performance (fat nodes) get a bigger role in order to push down latencies.

Last but not least, as far as I am aware, CG does not benefit from the recently introduced tensor cores in Nvidia’s Volta generation. (Simply because those cores are also made for dense matrix-matrix operations and packing overhead and reduction of parallelism in blocks does not play well when also calculating a lot of zeros.) LINPACK, or to be more precise the multi-node implementation HPL, does not use tensor cores yet either, but research is done in that field.

So all-in-all, HPCG as the Flop/s measure for an energy comparison seems to be an exciting and somewhat fair starting point to look at.

Method for an (Unofficial) “Green HPCG”

So let us take the official HPCG list and calculate a “Green HPCG” from it. Why should LINPACK be the only metric for Flop/s and efficiency? This “unofficial” list is just like the official Green500 list, but swapping LINPACK Flop/s for HPCG Flop/s.

Of course, data was not complete: not all HPCG (and even TOP500) candidates submit power and not all Green 500 listings have a HPCG submission.

And the Winner is …

MareNostrum P9 CTE with 327 MFlop/Ws! A system at rank 411 in the TOP500! Positioned slightly before AI Bridging Cloud (309 MFlop/Ws) and Summit (299 MFlop/Ws).

Sierra is on place four with 241 MFlop/Ws - interesting difference for a Summit-twin.

The list goes on with recent GPU systems until Xeon Phi (MIC) kicks in at 142 MFlop/Ws for Oakforest-PACS and the best x86 system, BSC with MareNostrum again (!), with 75 MFlop/Ws.

HPCG Rank	LINPACK Rank	System	GFlop/Ws	Arch	Launch
54	411	MareNostrum P9 CTE- IBM Power System AC922 IBM...	0.326807	GPU	06/2018
5	7	AI Bridging Cloud Infrastructure (ABCI)- PRIME...	0.308534	GPU	06/2018
1	1	Summit- IBM Power System AC922 IBM POWER9 22C ...	0.299065	GPU	06/2018
2	2	Sierra- IBM Power System S922LC IBM POWER9 22C...	0.241418	GPU	06/2018
16	22	TSUBAME3,0- SGI ICE XA IP139-SXM2 Xeon E5-2680...	0.238133	GPU	06/2017
6	5	Piz Daint- Cray XC50 Xeon E5-2690v3 12C 2,6GHz...	0.208444	GPU	11/2016
14	15	HPC4- Proliant DL380 Gen10 Xeon Platinum 8160 ...	0.205962	GPU	06/2018
35	51	DGX Saturn V- NVIDIA DGX-1 Xeon E5-2698v4 20C ...	0.192904	GPU	11/2016
9	14	Oakforest-PACS- PRIMERGY CX1640 M1 Intel Xeon ...	0.141788	MIC	11/2016
48	243	JOLIOT-CURIE KNL- Bull Sequana X1000 Intel Xeo...	0.114417	MIC	06/2018
11	16	Tera-1000-2- Bull Sequana X1000 Intel Xeon Phi...	0.105022	MIC	06/2018
10	12	Cori- Cray XC40 Intel Xeon Phi 7250 68C 1,4GHz...	0.090236	MIC	11/2016
21	25	MareNostrum- Lenovo SD530 Xeon Platinum 8160 2...	0.074902	x86	06/2017
49	91	ATERUI II - Cray XC50 Xeon Gold 6148 20C 2,4GH...	0.073892	x86	06/2018
4	6	Trinity- Cray XC40 Xeon E5-2698v3 16C 2,3GHz I...	0.072066	MIC	11/2017
23	56	SORA-MA- Fujitsu PRIMEHPC FX100 SPARC64 XIfx 3...	0.066667	SPARC64	06/2015
24	33	Electra- HPE SGI 8600/SGI ICE-X E5-2680V4/ Xeo...	0.063201	x86	11/2018
26	63	Fujitsu PRIMEHPC FX100 SPARC64 XIfx 32C 2,2GHz...	0.062594	SPARC64	11/2015
31	84	Plasma Simulator- Fujitsu PRIMEHPC FX100 SPARC...	0.058843	SPARC64	06/2015
41	40	JOLIOT-CURIE SKL- Bull Sequana X1000 Xeon Plat...	0.057448	x86	06/2018
57	361	Falcon- SGI ICE X Xeon E5-2695v4 18C 2,1GHz In...	0.055593	x86	11/2017
32	69	TSUBAME 2,5- Cluster Platform SL390s G7 Xeon X...	0.051859	GPU	06/2014
34	44	JURECA- T-Platforms V-Class/Dell C6320P E5-268...	0.050800	GPU	11/2017
25	36	Cheyenne- SGI ICE XA Xeon E5-2697v4 18C 2,3GHz...	0.050539	x86	11/2016
46	131	Prometheus- HP Apollo 8000 Xeon E5-2680v3 12C ...	0.049455	GPU	11/2015
40	253	iDataPlex DX360M4 Intel Xeon E5-2680v2 10C 2,8...	0.048786	x86	11/2013
3	18	K computer SPARC64 VIIIfx 2,0GHz Tofu intercon...	0.047610	SPARC64	11/2011
55	246	Cobalt- bullx DLC 720 Xeon E5-2680v4 14C 2,4GH...	0.043080	x86	06/2016
50	90	Beaufix2- bullx DLC 720 Xeon E5-2698v4 20C 2,2...	0.042580	x86	11/2016
51	89	Prolix2- bullx DLC 720 Xeon E5-2698v4 20C 2,2G...	0.042570	x86	06/2016
18	21	Mira- BlueGene/Q Power BQC 16C 1,60GHz Custom ...	0.042345	x86	06/2012
12	10	Sequoia- BlueGene/Q Power BQC 16C 1,60 GHz Cus...	0.041872	x86	06/2012
28	39	Vulcan- BlueGene/Q Power BQC 16C 1,600GHz Cust...	0.041019	x86	06/2013
22	32	Shaheen II- Cray XC40 Xeon E5-2698v3 16C 2,3GH...	0.040180	x86	05/2015
43	55	HPC2- iDataPlex DX360M4 Intel Xeon E5-2680v2 1...	0.039853	GPU	06/2014
17	27	Pleiades- SGI ICE X Intel Xeon E5-2670/E5-2680...	0.039750	x86	11/2016
56	271	Graham- Huawei X6800 V3 Xeon E5-2683 v4 16C 2,...	0.039539	GPU	06/2017
45	62	Mistral- bullx DLC 720 Xeon E5-2680v3 12C 2,5G...	0.039525	x86	06/2016
13	9	Titan- Cray XK7 Opteron 6274 16C 2,200GHz Cray...	0.039264	GPU	11/2012
19	34	Pangea- SGI ICE X Xeon Xeon E5-2670/ E5-2680v3...	0.039202	x86	06/2016
20	30	Hazel Hen- Cray XC40 Xeon E5-2680v3 12C 2,5GHz...	0.038174	x86	11/2014
47	92	Tianhe-2 LvLiang Solution- Tianhe-2 LvLiang In...	0.037723	MIC	11/2014
38	42	Cray XC40 Xeon E5-2695v4 18C 2,1GHz Aries inte...	0.034507	x86	06/2016
37	43	Cray XC40 Xeon E5-2695v4 18C 2,1GHz Aries inte...	0.034507	x86	06/2016
52	105	Tera-1000-1- bullx DLC 720 Xeon E5-2698v3 16C ...	0.033772	x86	06/2016
44	77	occigen2- bullx DLC 720 Xeon E5-2690v4 14C 2,6...	0.031797	x86	06/2017
7	3	Sunway TaihuLight- Sunway MPP Sunway SW26010 2...	0.031283	Sunway	06/2016
59	230	Sid- bullx DLC 720 Xeon E5-2695v4 18C 2,1GHz I...	0.030718	x86	06/2016
58	360	Cartesius 2- Bullx DLC B710/B720 Blades Xeon E...	0.027578	x86	06/2015
60	212	Endeavor- Intel Cluster Intel Xeon Gold 6148/X...	0.025893	MIC	11/2017
29	186	ARCHER- Cray XC30 Intel Xeon E5 v2 12C 2,700GH...	0.024436	x86	11/2014
27	64	SuperMUC- iDataPlex DX360M4 Xeon E5-2680 8C 2,...	0.024338	x86	06/2012
42	231	Curie thin nodes- Bullx B510 Xeon E5-2680 8C 2...	0.023917	x86	06/2012
30	134	Edison- Cray XC30 Intel Xeon E5-2695v2 12C 2,4...	0.020987	x86	06/2014
61	416	EAGLE- Huawei E9000 Blade Server Xeon E5-2697v...	0.015520	x86	11/2015
62	487	Lomonosov- T-Platforms T-Blade2/1,1 Xeon X5570...	0.000621	x86	06/2011

Notes: Some systems are mixed, e.g. JURECA (11/2017 upgrade) has Kepler GPUs (2015) and Xeon Phi accelerators (2017), which is categorized as “GPU” arch (probably wrong categorization and likely benchmarked for a sub-system). For SuperMUC-NG, no official power-numbers are yet available, maybe 3-6 MW are likely (estimating 69-35 MFlop/Ws).

Trends

The top places are taken by Nvidia P100 & V100 GPUs, likely due to two reasons: a significant efficiency improvement in Pascal and Volta and (finally) the launch of multi-accelerator nodes in large systems.

(Karl, if you are reading this: let’s put new numbers in your post to falsify or proof the first point ;-) )

On the second group is taken by (2017) MIC systems. I tried to group by “launch” of systems in TOP500 submissions, looking at an evolution over time. One can see that there was already a significant gap in 2016/17 between the two which has now been widened with Knights Hill being canceled and Knights Mill (likely also a LINPACK/ML focused hardware) not yet released.

As a little surprise, the RISC architecture SPARC64 which launched in year 2011 is still amazingly well positioned. Even today, it is close to recently built x86 systems. Let’s see where the upcoming A64FX will be heading.

Sunway’s SW26010, introduced in 2016, is surprisingly low in our little comparison.

Conclusions

General purpose RISC architectures are leading in this comparison even more than in the official Green500. In that sense, it would be really interesting if a power consumption number was submitted for Sandia’s Astra (Cavium ThunderX2) cluster. But either way, is our comparison fair?

Likely not more than any other metric for calculating efficiency. But let’s assume that most scientific HPC applications lie somewhere between the official LINPACK and HPCG benchmarks, so we should consider both metrics for efficiency as well.

Data

An interactive Jupyter Notebook and the HPCG data with added system power consumption have been uploaded to DOI:10.14278/rodare.68

Enjoy playing with it!

Updates

2018-11-18 3:38PM (CET): Avoid implying that HPL is able to use FP16 tensor cores already for TOP500 benchmarks. Clarified with an additional sentence and link to a recent paper, thanks Timothy!

reMarkable .lines File Format

2017-12-26T14:36:12+01:00

Happy holidays everyone!

The days before Christmas I took two evenings for a little project. I tried to understand the binary file format the reMarkable e-ink tablet creates when drawing on it, the .lines (nowadays .rm) file.

.rm files are part of the stored notebooks (and annotated PDFs) on the tablet. It is stored together with further textual meta information (JSON) under .local/share/remarkable/xochitl/.

Since this is my first time decoding a binary format my approach might have been a bit… unusual. Anyway, this blog post will explain you what I found and a follow-up post will explain the how (cliffhanger!).

Notebooks

New and annotated files on the reMarkable tablet are called notebooks. A notebook consists of pages. Each page has one background layer (PNG or PDF page) and up to five layers for your drawings. Each layer then stores a set of lines which again consists of points (dots? ;) ).

(Note: I am defining the wordings as I go to names that appear reasonable to me.)

A single line can have exactly one color, one brush type & one base size. Each point inside a line has three attributes: two coordinates, a pressure and two angles for the pen’s steepness to the surface.

.rm Format

reMarkable .lines (now .rm) files are contiguous, uncompressed binaries with sections similar to the RAM representation of a plain struct (POD). The byte order is little endian and standard data types are used (ASCII chars, four-Byte ints and IEEE floats).

The first 43 Bytes consist of the char-text reMarkable lines with selections and layers and zero-padding. Update v3: now reMarkable .lines file, version=3 and 10 spaces.

If follows an int32_t (4 Bytes) with the number of pages. (Removed in v3, now one page per file.) After that, loop over the retrieved number of pages - the file format contains no end-of-page, end-of-file or similar symbols.

Directly at the beginning of the now following page section, the next 4 Bytes are an int32_t again for the number of layers inside the page. Again, loop over the retrieved number of layers for the next reads.

A layer then in turn has first 4 Bytes with an int32_t for the number of lines inside it. You now guessed it, loop over the retrieved number of lines for the next reads.

Now a line has the following attributes: 4 Bytes int32_t for the brush type, 4 Bytes int32_t for the color of the line, and an other 4 Bytes “padding” which in all my files is zero. Then follows the base brush size in 4 Bytes float32_t aaaannd the number of points inside the current line as int32_t (4 Bytes).

Deducting from the file header ... with selections ..., I would bet the always-zero padding attribute should likely correspond to selected (yes/no) from the selection tool. The selection tool (🜊) in the tablet’s GUI allows to select a list of complete lines from an intersection with a selected free form for move/scale/duplicate operations. Anyway, this value is in all my files serialized as zero (maybe a bug) which corresponds to the observation that between closing and opening a file selections are not restored.

Finally, points: You guessed it again, reading the next block of points needs to be done in a loop as retrieved in the number of points attribute of the current line. A point consists of five times 4 Bytes of float32_t in the order of: X coordinate, Y coordinate, pen pressure, pen rotation to X axis, pen rotation to Y axis.

Update v3: now x, y, speed, direction, width, and pressure.

… and that is it!

Magic Numbers & Ranges

The integers in the file format for types can take certain magic numbers mapping them to implemented functionality. Floating point attributes have ranges (and units).

Line Attributes

The display seems to support a single shade of gray and black. Since lines can be drawn on top of each other, this makes three colors (including white lines):

The brush types selectable in the GUI are represented as:

While the base brush size can vary between the following values:

Point Attributes

Remarkably, coordinates are stored as float32_t instead of a possible representation in int (e.g. even int16_t!). This probably just saves some type conversion during the rendering or might originate from the representation in the touch driver. Nevertheless, from the values I saw it does not look too likely that the touch resolution is higher than the display resolution. Coordinates are withing the range of the resolution of the display: 0.0 to 1404. for X and 0.0 to 1872. for Y:

The pressure of drawing a certain point can vary between zero and one:

The pen rotation is the most interesting attribute. As in the sketch below, the rotation is given in Radian to the normal of the page for both the angle to X and Y axis. The range can be in principle from minus Pi half to Pi half, although the maximum will unlikely be reached as the pen tip will not touch the surface if you try to hold it parallel to the surface ;)
Input validation fans take care: if a user rams the pen from the backside through the tablet and manages to draw a point with it, the full mathematical range from minus Pi to Pi for this input can be reached!

Here is again the full cheat sheet for your enum implementations :)

From Pen to Paper

Now that we understand how each and every stroke on our tablet is stored we can ask the question: What defines a rendered line?

Very similar to Gimp’s interface of drawing lines with pens, a rendered representation will now place “brushes” on each dot. Brushes are partly-transparent PNGs that are placed on top of each other. I guess the reMarkable is writing points with a constant sampling rate, which will make slowly drawn lines look continuous while fast strokes are reveling the beautiful structure of the selected brush type (try it with a nice big size!).

The interesting parts: of course, the brush PNG needs to be oriented during rendering. Contrary to what one would guess for e.g. a stylograph (not implemented in the reMarkable), these are not rotated according to the pen’s rotation during the draw but to the tangents of two consecutive points during draw. This gives a paint brush like orientation of the rendered structure of lines - for all available brushes…

The actual width of a line assembles from four attributes. Besides the brush type (PNG template) the pressure and the rotation to the surface determine the width of a line.

Fun fact: The latter could actually be represented with only one attribute for the currently implemented brushes… so maybe we will see a calligraphy pen some time via an update? And by the way, what does brush type 6 stand for? :)

Fun fact 2: if you export a stroke via PDF (currently 1.4) the rotation of the PNG brushes is broken and always appears to be zero in all readers I tried (reported; seen in Gimp, Inkscape, Evince, Okular, …).

What to Do Next?

We do now understand our own data, our very own files we wrote with our reMarkable tablet. So what’s next and why to understand it anyway?

For easier handling for the format, we should define its formal grammar in a more sophisticated way than my writing above. As good open source contributors, we can also add recognition of the file format to the file command:

Now:

$ file myDrawing.rm
myDrawing.rm: data

Then:

$ file myDrawing.rm
myDrawing.rm: reMarkable tablet page (v3), 1404 x 1872, 4 layer(s)

Update 2018-07-25: Just put this in your $HOME/.magic and it will add file recognition!
Update 2018-07-26: Aaaaand… it’s mainline now, coming with file v5.35 :-)
Update 2019-02-09: Updated to version 3

0       string  reMarkable
>11     string  .lines
>>18    string  file,
>>>24   string  version=
>>>>32  byte    x       reMarkable tablet page (v%c), 1404 x 1872,
>>>>>43 lelong  x       %d layer(s)

We of course need an API in your favorite language for reading & writing .rm files. On top of such, we can write converters to render .rm files, e.g. allowing to change brush types in post processing to add a stylograph brush type for writing as with fancy dip pens :)

Update 2017-12-29: Hey let’s just start right away. Here is my little C++ file API and some first render examples to PNG: lines-are-beautiful
Update 2018-07-29: Also supports writing .lines (.rm) files now.

Also, we could with such converters improve exports - e.g. an exported PDF of the reMarkable tablet is easily 20 MB in size with the current vendor implementation and contains the bug described above.

And we could add our own cloud backends, e.g. to upload the small binary files to our own Nextcloud instances.

You have more ideas or are already implementing some of the above? Feel free to share it as well! :) (See my public contact details in the footer of the page.)

Updates

Just after publishing I found this reddit. It seems the missing 4 Bytes could indeed be something from the select-and-transform tool. Let’s see if we can get a non-zero value in there :)

2017-12-29: I just got note that the reMarkable wiki and the reHackable GitHub organization are also really far in their converters and file format documentations, including a working .lines to svg converter for all strokes on the device! Kudos, that’s great work everyone! :)

2018-10-30: The latest update just announced hand-writing detection, improved PDF export and beta SVG export :-)

2018-09-24: I drafted an implementation in Rust: lines-are-rusty

2019-02-09: The .lines format has slightly changed to version 3 end of 2018. Notebooks are now stored as one file per page, the file ending is .rm and the points attributes are now: x, y, speed, direction, width, and pressure. See this update (or that one).

Referral Program

On Oct 4th, 2018 reMarkable started a program to get 85 Euro off and me 85 Euro cashback. So let’s give this a try, here is my referrer link: buy a reMarkable tablet (affiliate link)

Version

“Codex”: 0.0.4.81

/etc/version: 20170911122159

$ file xochitl
xochitl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0f8759a2e5ad62bb4f16d3b64a54e2bbef0ab271, stripped

Exploring the reMarkable via Shell

2017-12-18T01:53:06+01:00

[Privacy]

I am using (and recommending) my reMarkable tablet for a few months now. The display is just lovely, crisp and responsive and I am one of the few lucky ones receiving it in the first badges that came out.

As a good user, I am not complaining (often) publicly about the bugs I stumbled up (and reported) and expect most of them to be fixed soon, even after 3 months without any software updates seen. Just be aware that convenient data import/export is still a big problem for Android + Linux -only users, you will have to stick to plugging in a USB cable and using the (beta) web app.

Anyway, one of the exciting features of the reMarkable is that it comes with a full running Linux (4.1.28) and enabled root SSH access - thanks to the GPL :)

Wuhu, nice! #GPL #Iamroot @remarkablepaper pic.twitter.com/XIJOQU4ak5
— Axel Huebl (@axccl) October 10, 2017

remarkable: ~/ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 48.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
Hardware : Freescale i.MX6 SoloLite (Device Tree)

remarkable: ~/ uname -a
Linux remarkable 4.1.28-zero-gravitas-01897-g7f82abb869aa-dirty #256 Tue Sep 12 08:53:59 CEST 2017 armv7l GNU/Linux

This blog post is exploring what we can do beyond the GUI-facing part of the device (e.g. to fix bugs and add missing features).

I am not trying to reverse engineer the features in hardware and software that make this tablet superior to anything I saw on the market. You can read more on that on HN by their CTO, which also contributed in the past to FOSS projects such as KDE.

My primary interest is more usable data exchange and (way better) data archiving & meta data of my (scientific) work on it. On a personal side node, I think it would increase the reMarkable’s market value if a documented API for efficient drawing and open file exchange (both accessing to their cloud and documenting the internal file formats) would be made public. Especially the last aspect should be better sooner than later, since the created data of the user (hopefully is) and should never be relevant for their successful hardware project.

Note: This blog post might be updated on the go in the future.

Home Structure

If one first logs in one finds oneself as uid=0 (root) on /home/root. The first things to hop on are the directories and config files within it, cat, less and vim are pre-installed.

.config/ for xochitl

wifi pws
serial number
settings from UI
ssh login
left/right handed
last emails used for “share”
device token

.local/share/remarkable/xochitl/

Individual PDFs, epubs(?) and reMarkable notebooks

remarkable: ~/ ls /home/root/.local/share/remarkable/xochitl/
f434bd89-64ec-4071-ad75-806e5a951d93.cache/
f434bd89-64ec-4071-ad75-806e5a951d93.content
f434bd89-64ec-4071-ad75-806e5a951d93.highlights/
f434bd89-64ec-4071-ad75-806e5a951d93.lines
f434bd89-64ec-4071-ad75-806e5a951d93.lines.backup
f434bd89-64ec-4071-ad75-806e5a951d93.metadata
f434bd89-64ec-4071-ad75-806e5a951d93.pagedata
f434bd89-64ec-4071-ad75-806e5a951d93.thumbnails/

f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.cache/
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.content
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.highlights/
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.lines
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.lines.backup
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.metadata
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.pagedata
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.pdf
f4e2bd4e-c533-4fd0-a868-d63a1ff1ab1d.thumbnails/

Besides some json meta files, most of the files are some binary format. Understanding these would be wonderful to truly control ones data! Unfortunately, I have not seen anyone trying to document it yet.

Update December 19th: ok, I decoded the .lines file now :) A blog post will follow!
Update December 26th: and here the .lines format is!

And nearly done. Layers identified and read :) pic.twitter.com/jeJSIzjN3H
— Axel Huebl (@axccl) December 19, 2017

Last but not least, one can also add the usual Linux stuff. For example just add your key to .ssh/authorized_keys for pwless login.

Wifi Beyond the GUI

By default, connecting to a WPA-EAP network (“WPA-Enterprise”) such as Eduroam is not possible throughout the GUI. This is especially bad for academic users. Since wifi is controlled via the well-established “wpa supplicant” tool let us change this!

Append to /etc/wpa_supplicant/wpa_supplicant-wlan0.conf (example for TU Dresden):

network={
    ssid="eduroam"
    key_mgmt=WPA-EAP
    eap=PEAP
    identity="sNumber@tu-dresden.de"
    anonymous_identity="anonymous@tu-dresden.de"
    password="YourPasswordHere"
    ca_cert="/etc/wpa_supplicant/TUD-CACert.pem"
    phase2="auth=MSCHAPV2"
}

Add command line option to /lib/systemd/system/wpa_supplicant@.service:

[Service]
Type=simple
ExecStart=/usr/sbin/wpa_supplicant -C/var/run/wpa_supplicant -i%I

[Service]
Type=simple
ExecStart=/usr/sbin/wpa_supplicant -C/var/run/wpa_supplicant -i%I -c /etc/wpa_supplican
t/wpa_supplicant-wlan0.conf

It still does not auto-connect yet, but if you restart wpa_supplicant it connects:

systemctl daemon-reload
systemctl restart wpa_supplicant@wlan0.service

https://tu-dresden.de/zih/dienste/service-katalog/arbeitsumgebung/zugang_datennetz
https://www.lrz.de/services/netz/mobil/802_1x/802_1x-linux/
https://www.cms.hu-berlin.de/de/dl/netze/wlan/config/eduroam/linux

Templates

The reMarkable supports drawing in up to five layers on PDFs and creating “notebooks” with individual “templates” as background. Templates are straight PNGs parsed from a directory and the filename.

remarkable: ~/ ls /usr/share/remarkable/templates/
Blank.png                 LS Week US.png            P Grid top.png
Isometric.png             LS Week.png               P Lined bottom.png
LS Checklist double.png   Notes.png                 P Lined heading.png
LS Checklist.png          P Black dots.png          P Lined top.png
LS Dayplanner.png         P Black grid.png          P Lines large.png
LS Dots bottom.png        P Black lines.png         P Lines medium.png
LS Dots top.png           P Black.png               P Lines small.png
LS Four storyboards.png   P Checklist.png           P Margin large.png
LS Grid bottom.png        P Cornell.png             P Margin medium.png
LS Grid margin large.png  P Day.png                 P Margin small.png
LS Grid margin med.png    P Dots S bottom.png       P One storyboard.png
LS Grid top.png           P Dots S top.png          P Two storyboards.png
LS Lines bottom.png       P Dots S.png              P US College.png
LS Lines medium.png       P Dots large.png          P US Legal.png
LS Lines small.png        P Four storyboards.png    P Week 2.png
LS Lines top.png          P Grid bottom.png         P Week US.png
LS Margin medium.png      P Grid large.png          P Week.png
LS Margin small.png       P Grid margin large.png   Perspective1.png
LS One storyboard 2.png   P Grid margin med.png     Perspective2.png
LS One storyboard.png     P Grid medium.png
LS Two storyboards.png    P Grid small.png

PNG image data, 1404 x 1873, 8-bit/color RGBA, non-interlaced

By adding two additional files I successfully added two new templates. I added two landscape variants of the portrait Notes.png with -33% and -50% line height.

One good further thing could be a bullet journal template (if I ever understand how it actually works.) What the template selection needs in general are thumbnails for faster loading and grouping by topic, which we can not influence at this point. It is also worth noting that one can adjust the splash, sleep and empty battery screens under /usr/share/remarkable-shutdown.

xochitl

This is the main app of the reMarkable, which is started by systemd. Like all the (user-facing) software related to the product it is written in Qt 5.9. Under remarkable.engineering you can even find a pre-bundled SDK to develop your own apps.

Realistically, you can already build and run other Qt apps that view things, but drawing without reusing their own specially-crafted low-latency API will not be fun.

Anyway, if one likes to dig deeper: checking the shipped libraries for Qt5 might be worth investigating for APIs in case they never will be documented:

remarkable: ~/ ls /usr/lib/qt5/plugins/platforms/
libepaper.so      libqlinuxfb.so    libqminimal.so    libqoffscreen.so

From a first quick analyze of the app via strace it might be a bit suboptimal that xochitl calls clock_gettime a dozen times a second and tries to access quite a few not-shipped library files during its usual execution. Room for improvement I guess, especially for battery lifetime.

Web UI (Beta)

Allows to download all files as PDF and upload new files (PDF, epub) by just drag-and-dropping on the browser window. This functionality is also part of xochitl the same way as the file conversion is. After enabling it in the settings page (rM - Storage - Enable USB web interface) and DHCP-ing the new USB ethN device it is reachable at http://10.11.99.1/

I was always wondering why the PDFs via share and download are so extremely large for simple drawn notebook pages. It turns out it is likely that the conversion to PDF is still rather hard-coded (strace again) and uncompressed:

# many lines
1043  write(35, "<<\n/Type /Pattern\n/PatternType 1"..., 16168) = 16168

# few lines
1043  write(35, "/Pat24782 24782 0 R\n/Pat24783 24"..., 16380) = 16380
1043  write(35, "/Pat25601 25601 0 R\n/Pat25602 25"..., 16380) = 16380

This explains the dramatic size of the created PDFs (easily a dozen MB for a sketch). It would actually be very nice if a cmd line tool and/or API exists for those conversions as well.

Other apps

Other apps running by default (from /usr/bin) are

update_engine (relatively quiet)
crashuploader in /home/root (relatively active)

Of course, you never agreed to the crash collection and report uploading beforehand. Ouch.

Update 2018-01-05: Asked the support about crashuploader. They responded that “[t]he data collected contains no identifying data” and can be disabled in systemd via systemctl disable crashuploader && systemctl stop crashuploader.

Without analyzing the network traffic yet, it is still calming to see a public key in /usr/share/update_engine/update-payload-key.pub.pem.

Cross Compile

I did not use the SDK on their homepage yet but had good success using dockcross for linux-armv7. Compiling powertop with it two months ago I still fund no missing tunable besides VM writeback timeout to improve battery life.

FROM dockcross/base:latest

# Add the cross compiler sources
RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
  dpkg --add-architecture armhf && \
  curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -

RUN apt-get update && apt-get install -y \
  crossbuild-essential-armhf \
  gfortran-arm-linux-gnueabihf \
  qemu-user \
  qemu-user-static \
  libbz2-dev:armhf \
  libexpat1-dev:armhf \
  ncurses-dev:armhf \
  libssl-dev:armhf \
  libnl-3-dev:armhf \
  libnl-genl-3-dev:armhf \
  libnl-utils:armhf \
  libpci-dev:armhf \
  pkg-config:armhf \
  autoconf \
  autogen \
  autotools-dev \
  automake \
  libtool \
  autopoint \
  gettext

ENV CROSS_TRIPLE arm-linux-gnueabihf
ENV CROSS_ROOT /usr/${CROSS_TRIPLE}
ENV AS=/usr/bin/${CROSS_TRIPLE}-as \
    AR=/usr/bin/${CROSS_TRIPLE}-ar \
    CC=/usr/bin/${CROSS_TRIPLE}-gcc \
    CPP=/usr/bin/${CROSS_TRIPLE}-cpp-4.9 \
    CXX=/usr/bin/${CROSS_TRIPLE}-g++ \
    LD=/usr/bin/${CROSS_TRIPLE}-ld

# Tuned for reMarkable tablet:
#   Freescale i.MX6 SoloLite
#   ARM A9 CPU @1GHz (1 Core, Cortex A9)
#   model name : ARMv7 Processor rev 10 (v7l)
#   BogoMIPS : 48.00
#   Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
# evaluate:
#   -mslow-flash-data
#   -mfloat-abi=hard
ENV CFLAGS="-march=armv7-a -mtune=cortex-a9 -mfpu=neon -funsafe-math-optimizations -mthumb" \
  CXXFLAGS="-march=armv7-a -mtune=cortex-a9 -mfpu=neon -funsafe-math-optimizations -mthumb"

#ENV DEFAULT_DOCKCROSS_IMAGE dockcross/linux-armv7
ENV DEFAULT_DOCKCROSS_IMAGE linux-armv7

# Note: Toolchain file support is currently in debian Experimental:
# https://wiki.debian.org/CrossToolchains#In_jessie_.28Debian_8.29
COPY Toolchain.cmake /usr/lib/${CROSS_TRIPLE}/
ENV CMAKE_TOOLCHAIN_FILE /usr/lib/${CROSS_TRIPLE}/Toolchain.cmake

# build examples
#   powertop
#   ./autogen.sh
#   ./configure LIBNL_CFLAGS="-I/usr/lib/arm-linux-gnueabihf/libnl3 -I/usr/include/libnl3" LIBNL_LIBS="-L/usr/lib/arm-linux-gnueabihf -lnl-3 -lnl-genl-3"
#   make

Building more tools, it turns out that just installing a cross-compiler for ARM and a few multi-arch libs on Debian/Ubuntu is way easier:

# add multi-arch for "armhf"
# https://wiki.debian.org/Multiarch/HOWTO
dpkg --add-architecture armhf
apt-get update

# install C/C++ compiler and standard libs
apt-get install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf
apt-get install libstdc++6:armhf libelf-dev:armhf

# compile something
arm-linux-gnueabihf-g++ main.cpp

Then copy over the created executable (and the dynamic libraries from the armhf packages above which it might be missing during execution) to your reMarkable via scp.

Referral Program

On Oct 4th, 2018 reMarkable started a program to get 85 Euro off and me 85 Euro cashback. So let’s give this a try, here is my referrer link: buy a reMarkable tablet (affiliate link)

Version

“Codex”: 0.0.4.81

/etc/version: 20170911122159

$ file xochitl
xochitl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0f8759a2e5ad62bb4f16d3b64a54e2bbef0ab271, stripped