Potential spiped 1.4.0 release -- please test

[Colin Percival]

Hi all,

I've attached a tarball for what might be spiped 1.4.0 -- depending on whether
anyone reports any problems with it. There are significant changes in the
build code, so please test -- even if all you have time to do is confirm that
it builds, that's better than nothing.

In particular I'd like to see this tested on as many different compilers as
possible and on different (non-x86) platforms, since this new version contains
code for detecting and using AESNI instructions -- the build code *should*
disable this on non-x86 platforms, but I'd like some confirmation.

Changes since 1.3.1:
* -g option causes spipe and spiped to drop connections if the other host is
using the -f (fast handshake, no forward secrecy) option.
* Automatic detection and use of AESNI instructions.
* lots and lots of minor code cleanups.

Hash of attached tarball:
SHA256 (spiped-1.4.0a.tgz) =
c54827ffba2f8e1deff9a92c13d61ec9bd0b2ae582b7a165f4e275bcd686ef97

- --
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

[Dmitry Chestnykh]

Hello,

Here's what I get on Xubuntu 14.04 with Atom CPU (doesn't support AESNI):


~/sources/spiped-1.4.0a $ make
export CFLAGS="${CFLAGS:--O2}" \
export LDADD_POSIX=`export CC="cc"; cd POSIX && command -p sh
posix-l.sh`; \
export CFLAGS_POSIX=`export CC="cc"; cd POSIX && command -p sh
posix-cflags.sh`; \
( export CC="cc"; cd libcperciva/cpusupport/Build && command -p sh
cpusupport.sh ) > cpusupport-config.h; \
. cpusupport-config.h; \
for D in spiped spipe; do \
( cd ${D} && make all ) || exit 2; \
done
WARNING: POSIX violation: make's CC doesn't understand -lxnet
Checking if compiler supports X86 CPUID feature... yes
Checking if compiler supports X86 AESNI feature... yes, via -maes
/bin/sh: 5: .: cpusupport-config.h: not found
make: *** [all] Error 2


===================

cpusupport-config.h is here:

~/sources/spiped-1.4.0a $ cat cpusupport-config.h
#define CPUSUPPORT_X86_CPUID
#define CPUSUPPORT_X86_AESNI
#ifdef cpusupport_dummy
export CFLAGS_X86_AESNI="-maes"
#endif


$ uname -a
Linux dchest-samsung 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13
15:45:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ gcc --version
gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
--
Dmitry Chestnykh
http://www.codingrobots.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
I've attached a tarball for what might be spiped 1.4.0 -- depending on whether
anyone reports any problems with it. There are significant changes in the
build code, so please test -- even if all you have time to do is confirm that
it builds, that's better than nothing.
In particular I'd like to see this tested on as many different compilers as
possible and on different (non-x86) platforms, since this new version contains
code for detecting and using AESNI instructions -- the build code *should*
disable this on non-x86 platforms, but I'd like some confirmation.
* -g option causes spipe and spiped to drop connections if the other host is
using the -f (fast handshake, no forward secrecy) option.
* Automatic detection and use of AESNI instructions.
* lots and lots of minor code cleanups.
SHA256 (spiped-1.4.0a.tgz) =
c54827ffba2f8e1deff9a92c13d61ec9bd0b2ae582b7a165f4e275bcd686ef97
- --
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (FreeBSD)
iEYEARECAAYFAlP4SsAACgkQOM7KaQxqam6hqQCeMTTKV2sMGoyt1c2emqvWyjTL
I7YAoJeYJ7dVJY0joSmKmIczsUMvDtCK
=o2cC
-----END PGP SIGNATURE-----

[Dmitry Chestnykh]

After changing

- . cpusupport-config.h; \
+ . ./cpusupport-config.h; \


it compiles successfully and spipe command works (didn't try spiped).

[Colin Percival]

Post by Dmitry Chestnykh
After changing
- . cpusupport-config.h; \
+ . ./cpusupport-config.h; \
it compiles successfully

Aha, of course. I noticed that requirement in the standard (POSIX says
that . searches $PATH, but not necessarily the current directory; BSD
sh searches both, which is why my testing missed this) but I forgot to
make the change.

Committed, thanks!

Post by Dmitry Chestnykh
and spipe command works (didn't try spiped).

I assume this is on the non-AESNI-supporting Atom CPU you mentioned?

[Dmitry Chestnykh]

Post by Colin Percival

Post by Dmitry Chestnykh
and spipe command works (didn't try spiped).

I assume this is on the non-AESNI-supporting Atom CPU you mentioned?

Yes. Hmm...

~/sources/spiped-1.4.0a/libcperciva/cpusupport/Build $ sh cpusupport.sh
Checking if compiler supports X86 CPUID feature... no
Checking if compiler supports X86 AESNI feature... no

However, when running make, it is:

Checking if compiler supports X86 CPUID feature... yes
Checking if compiler supports X86 AESNI feature... yes, via -maes

(full make output attached)

I verified that my CPU doesn't have AESNI by grepping /proc/cpuinfo for
it, and also:

~/sources/spiped-1.4.0a/libcperciva/cpusupport/Build $ cc -maes
cpusupport-X86-AESNI.c && ./a.out
Illegal instruction

But, as I said, spipe works: I use SSH-over-spipe to connect to my
hosts, and verified that it works with new binary:

ProxyCommand ~/sources/spiped-1.4.0a/spipe/spipe -t %h:922 -k
~/.ssh/spiped_%h_key

[Colin Percival]

Post by Dmitry Chestnykh

Post by Colin Percival

Post by Dmitry Chestnykh
and spipe command works (didn't try spiped).

I assume this is on the non-AESNI-supporting Atom CPU you mentioned?

Yes. Hmm...
~/sources/spiped-1.4.0a/libcperciva/cpusupport/Build $ sh cpusupport.sh
Checking if compiler supports X86 CPUID feature... no
Checking if compiler supports X86 AESNI feature... no

Yes, cpusupport.sh isn't intended to be run directly -- it needs ${CC} set
from the Makefile.

Post by Dmitry Chestnykh
Checking if compiler supports X86 CPUID feature... yes
Checking if compiler supports X86 AESNI feature... yes, via -maes

This is correct: It's checking for compiler support, not whether the current
CPU can run the code, since you might build on one system and run the binary
on another.

Post by Dmitry Chestnykh
I verified that my CPU doesn't have AESNI by grepping /proc/cpuinfo for it, and
~/sources/spiped-1.4.0a/libcperciva/cpusupport/Build $ cc -maes
cpusupport-X86-AESNI.c && ./a.out
Illegal instruction
But, as I said, spipe works: I use SSH-over-spipe to connect to my hosts, and
ProxyCommand ~/sources/spiped-1.4.0a/spipe/spipe -t %h:922 -k
~/.ssh/spiped_%h_key

Good, everything it working -- the build code is detecting that your compiler
supports AESNI, is building support for it into the spipe binary, but then
when spipe is run it is detecting that your CPU does not support AESNI and is
using the OpenSSL code instead.

[Dmitry Chestnykh]

Post by Colin Percival
This is correct: It's checking for compiler support, not whether the current
CPU can run the code, since you might build on one system and run the binary
on another.
Good, everything it working -- the build code is detecting that your compiler
supports AESNI, is building support for it into the spipe binary, but then
when spipe is run it is detecting that your CPU does not support AESNI and is
using the OpenSSL code instead.

Oh, cool -- I like it when the actual CPI feature support is detected
during runtime! Thanks.

[James Turner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
I've attached a tarball for what might be spiped 1.4.0 -- depending on whether
anyone reports any problems with it. There are significant changes in the
build code, so please test -- even if all you have time to do is confirm that
it builds, that's better than nothing.
In particular I'd like to see this tested on as many different compilers as
possible and on different (non-x86) platforms, since this new version contains
code for detecting and using AESNI instructions -- the build code *should*
disable this on non-x86 platforms, but I'd like some confirmation.

Compiles cleany on OpenBSD -current amd64 (what will be 5.6). With the
default system gcc 4.2.1 the CPUID and AESNI compiler features are not
supported.

When compiled with gcc 4.8.2 from ports, both are supported and here is
the resulting cpusupport-config.h:

#define CPUSUPPORT_X86_CPUID
#define CPUSUPPORT_X86_AESNI
#ifdef cpusupport_dummy
export CFLAGS_X86_AESNI="-maes"
#endif

Hopes this helps

[James Turner]

Post by James Turner

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
I've attached a tarball for what might be spiped 1.4.0 -- depending on whether
anyone reports any problems with it. There are significant changes in the
build code, so please test -- even if all you have time to do is confirm that
it builds, that's better than nothing.
In particular I'd like to see this tested on as many different compilers as
possible and on different (non-x86) platforms, since this new version contains
code for detecting and using AESNI instructions -- the build code *should*
disable this on non-x86 platforms, but I'd like some confirmation.

Compiles cleany on OpenBSD -current amd64 (what will be 5.6). With the
default system gcc 4.2.1 the CPUID and AESNI compiler features are not
supported.
When compiled with gcc 4.8.2 from ports, both are supported and here is
#define CPUSUPPORT_X86_CPUID
#define CPUSUPPORT_X86_AESNI
#ifdef cpusupport_dummy
export CFLAGS_X86_AESNI="-maes"
#endif
Hopes this helps

Oh and besides needing the patch Dmitry pointed out when testing this
from the OpenBSD ports infrastructure I needed the following patch for
make install.
--
James Turner

--- Makefile.orig Sat Aug 23 22:59:43 2014
+++ Makefile Sat Aug 23 22:59:55 2014
@@ -15,7 +15,7 @@ all:
done

install: all
- export BINDIR=$${BINDIR:-${BINDIR_DEFAULT}} \
+ export BINDIR=$${BINDIR:-${BINDIR_DEFAULT}}; \
for D in ${PROGS}; do \
( cd $${D} && make install ) || exit 2; \
done

[Colin Percival]

Post by James Turner

Post by James Turner
Compiles cleany on OpenBSD -current amd64 (what will be 5.6). With the
default system gcc 4.2.1 the CPUID and AESNI compiler features are not
supported.
When compiled with gcc 4.8.2 from ports, both are supported and here is
#define CPUSUPPORT_X86_CPUID
#define CPUSUPPORT_X86_AESNI
#ifdef cpusupport_dummy
export CFLAGS_X86_AESNI="-maes"
#endif

Perfect, that's what I was expecting.

Post by James Turner
Oh and besides needing the patch Dmitry pointed out when testing this
from the OpenBSD ports infrastructure I needed the following patch for
make install.

Right, I've already committed that fix.

[Max Afonov]

Builds fine on Android (a recent CyanogenMod 11 nightly build) using a
Debian testing userland in a chroot. Looks like cpusupport-config.h
comes out empty.

uname -a: Linux localhost 3.4.0-cyanogenmod-ge5bf64b #1 SMP PREEMPT
Wed Aug 20 23:25:59 PDT 2014 armv7l GNU/Linux

/proc/cpuinfo (the hardware is a Nexus 7 tablet):

Processor : ARMv7 Processor rev 0 (v7l)
processor : 0
BogoMIPS : 13.50

processor : 1
BogoMIPS : 13.50

processor : 2
BogoMIPS : 13.50

processor : 3
BogoMIPS : 13.50

Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
vfpv4 idiva idivt
CPU implementer : 0x51
CPU architecture: 7
CPU variant : 0x1
CPU part : 0x06f
CPU revision : 0

Hardware : QCT APQ8064 FLO
Revision : 0000
Serial : 0000000000000000

[Colin Percival]

Post by Max Afonov
Builds fine on Android (a recent CyanogenMod 11 nightly build) using a
Debian testing userland in a chroot. Looks like cpusupport-config.h
comes out empty.

Perfect, that's what should happen when compiling for systems which don't
have x86 instructions.

[Frederick Akalin]

Alas, my report comes too late to fix 1.4.0 for OS X 10.9.4, but I
have a few minor issues to report.

The first problem is with POSIX/posix-cflags.sh. Its lines that do
'echo "-D..."' prints trailing newlines, but if only one POSIX
violation is detected it turns out okay since that trailing newline
gets eaten (I'm guessing by the backticks). However, if both POSIX
violations are triggered, then a newline gets inserted into the middle
of CFLAGS_POSIX, which confuses make:

$ bsdmake
export CFLAGS="${CFLAGS:--O2}"; export LDADD_POSIX=`export CC="cc";
cd POSIX && command -p sh posix-l.sh`; export
CFLAGS_POSIX=`export CC="cc"; cd POSIX && command -p sh
posix-cflags.sh`; ( export CC="cc"; cd
libcperciva/cpusupport/Build && command -p sh cpusupport.sh ) >
cpusupport-config.h; . ./cpusupport-config.h;
for D in spiped spipe; do ( cd ${D} && make all ) ||
exit 2; done
WARNING: POSIX violation: make's CC doesn't understand -lrt
WARNING: POSIX violation: make's CC doesn't understand -lxnet
WARNING: POSIX violation: <sys/socket.h> not defining MSG_NOSIGNAL
WARNING: POSIX violation: <time.h> not defining CLOCK_REALTIME
Checking if compiler supports X86 CPUID feature... yes
Checking if compiler supports X86 AESNI feature... yes, via -maes
cc -O2 -DPOSIXFAIL_MSG_NOSIGNAL
clang: error: no input files
make: *** [main.o] Error 1
*** Error code 2
Stop in /private/tmp/spiped-1.4.0.

Since sh's echo doesn't support the -n switch, a solution is to use
printf instead, e.g.:

printf %s "-DPOSIXFAIL_MSG_NOSIGNAL "

This seems to work for me. (Note the trailing space before the closing quote.)

The second less serious problem is that the root Makefile assumes make
== bsdmake. This seems not to cause any problems currently, but I
suggest fixing it anyway, by replacing 'make' with '${MAKE}'.

Hope this helps.

-- Fred

Post by Colin Percival

Post by Max Afonov
Builds fine on Android (a recent CyanogenMod 11 nightly build) using a
Debian testing userland in a chroot. Looks like cpusupport-config.h
comes out empty.

Perfect, that's what should happen when compiling for systems which don't
have x86 instructions.
--
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

[Colin Percival]

Hi Frederick & list,

Post by Frederick Akalin
Alas, my report comes too late to fix 1.4.0 for OS X 10.9.4, but I
have a few minor issues to report.

I can always release a version 1.4.1. ;-)

Post by Frederick Akalin
The first problem is with POSIX/posix-cflags.sh. Its lines that do
'echo "-D..."' prints trailing newlines, but if only one POSIX
violation is detected it turns out okay since that trailing newline
gets eaten (I'm guessing by the backticks). However, if both POSIX
violations are triggered, then a newline gets inserted into the middle

Very interesting. I assumed that all of the \n characters would be eaten
and replaced by linear whitespace.

Post by Frederick Akalin
printf %s "-DPOSIXFAIL_MSG_NOSIGNAL "

Yes, printf is what I've used elsewhere for constructing lines in parts.

Post by Frederick Akalin
The second less serious problem is that the root Makefile assumes make
== bsdmake. This seems not to cause any problems currently, but I
suggest fixing it anyway, by replacing 'make' with '${MAKE}'.

I don't assume that make is bsdmake, only that it is something POSIX-compliant.
But you're right that I should use ${MAKE}, in case it isn't in the path. (I
don't know if anyone actually gets this right, but if make is not in the path
and is invoked via /path/to/make, then ${MAKE} should be "/path/to/make" and
not just "make".)

Can you try the attached patch?

[Dominyk Tiller]

Hi Colin,

Thanks for the email. I should probably get around to subscribing to
the list to make these things a little easier.

The patch did indeed do the job, and now compiles without issue. It
throws a few build warnings due to depreciated declarations but
there's nothing fatal and nothing that stops spiped working. Thanks
for the very swift response on this!

Dominyk

Sent from Thunderbird for OS X. My PGP public key is automatically
attached to this email.

Hi Dominyk,
In case you didn't see messages on the list -- can you try the
attached patch?
Colin Percival
-------- Original Message -------- Subject: Re: Potential spiped
1.4.0 release -- please test Date: Sat, 30 Aug 2014 12:17:08 -0700
Hi Frederick & list,

Post by Frederick Akalin
Alas, my report comes too late to fix 1.4.0 for OS X 10.9.4, but
I have a few minor issues to report.

I can always release a version 1.4.1. ;-)

Post by Frederick Akalin
The first problem is with POSIX/posix-cflags.sh. Its lines that
do 'echo "-D..."' prints trailing newlines, but if only one
POSIX violation is detected it turns out okay since that trailing
newline gets eaten (I'm guessing by the backticks). However, if
both POSIX violations are triggered, then a newline gets inserted

Very interesting. I assumed that all of the \n characters would be
eaten and replaced by linear whitespace.

Post by Frederick Akalin
printf %s "-DPOSIXFAIL_MSG_NOSIGNAL "

Yes, printf is what I've used elsewhere for constructing lines in parts.

Post by Frederick Akalin
The second less serious problem is that the root Makefile assumes
make == bsdmake. This seems not to cause any problems currently,
but I suggest fixing it anyway, by replacing 'make' with
'${MAKE}'.

I don't assume that make is bsdmake, only that it is something
POSIX-compliant. But you're right that I should use ${MAKE}, in
case it isn't in the path. (I don't know if anyone actually gets
this right, but if make is not in the path and is invoked via
/path/to/make, then ${MAKE} should be "/path/to/make" and not just
"make".)
Can you try the attached patch?

[Frederick Akalin]

Post by Colin Percival
I don't assume that make is bsdmake, only that it is something
POSIX-compliant.
But you're right that I should use ${MAKE}, in case it isn't in the path.
(I
don't know if anyone actually gets this right, but if make is not in the
path
and is invoked via /path/to/make, then ${MAKE} should be "/path/to/make"
and
not just "make".)

My mistake, I confused the release Makefile with the one used during normal
development (i.e., the one in the repository). I'm running into some
additional errors with that one -- will send out a separate post when I
figure them out.

Can you try the attached patch?
Works for me (with both make and bsdmake)! Thanks.

-- Fred

[Colin Percival]

Post by Colin Percival
I don't assume that make is bsdmake, only that it is something POSIX-compliant.
But you're right that I should use ${MAKE}, in case it isn't in the path. (I
don't know if anyone actually gets this right, but if make is not in the path
and is invoked via /path/to/make, then ${MAKE} should be "/path/to/make" and
not just "make".)
My mistake, I confused the release Makefile with the one used during normal
development (i.e., the one in the repository). I'm running into some additional
errors with that one -- will send out a separate post when I figure them out.

If you're not running FreeBSD, you probably won't be able to use the Makefiles
from the SVN tree. This is normal -- those Makefiles are there to make my life
easier as a developer, not with any expectation that they will be useful for
anyone else.

Post by Colin Percival
Can you try the attached patch?
Works for me (with both make and bsdmake)! Thanks.

Great, thanks!

[Frederick Akalin]

Post by Colin Percival
If you're not running FreeBSD, you probably won't be able to use the
Makefiles
from the SVN tree. This is normal -- those Makefiles are there to make my
life
easier as a developer, not with any expectation that they will be useful
for
anyone else.

Fair enough. However, some of the problems I'm running into are inherently
interesting and may be worth fixing anyway.

[Frederick Akalin]

Post by Colin Percival
In particular I'd like to see this tested on as many different compilers as
possible and on different (non-x86) platforms, since this new version contains
code for detecting and using AESNI instructions -- the build code *should*
disable this on non-x86 platforms, but I'd like some confirmation.

For completeness, spiped 1.4.0 builds with AESNI and no warnings on
Ubuntu 14.04 x64 (gcc 4.8.2).