I recently added a binary file as privdata (installed with File.hasPrivContent). When I propellor --dump the privdata on either Linux or macOS, the contents is correct. Spinning from Linux also works, but spinning from macOS installs a corrupt version of the file. The corruption looks like every valid UTF-8 sequence has been replaced with a single byte which is the lowest byte of the Unicode codepoint encoded by the sequence, so this must have something to do with encodings, but from staring at the source code I can't figure out what. The data from --dump is not corrupted which seems especially strange.
Sounds like a problem with sendPrivData, which writes it to a Handle that's connected to propellor on the host being spun.
Handles have an associated encoding, which comes from the locale settings. The char8 TextEncoding sounds like what you describe (code point modulo 256). hSetEncoding can change it.
Here's a patch you could try that prints out the encoding in use and tries to force utf8.
Just UTF-8in both cases and the corruption is not fixed. I think the problem may be on the receiving side? On macOS my$LC_CTYPEis set to"UTF-8"which is passed through by SSH but is an invalid locale on Linux. Runningenv LC_CTYPE=C.UTF-8 ./propellor --spin blahfixes the corruption.I tried this patch:
--- a/src/Propellor/Spin.hs +++ b/src/Propellor/Spin.hs @@ -181,6 +181,8 @@ getSshTarget target hst -- running the updateServer update :: Maybe HostName -> IO () update forhost = do + hPrint stderr =<< hGetEncoding stdin + hSetEncoding stdin utf8 whenM hasGitRepo $ req NeedRepoUrl repoUrlMarker setRepoUrlI get
Just ANSI_X3.4-1968from the remote side but unfortunately the corruption persists.Err, this patch:
Ah, good debugging!
The code that runs on the remote side is Propellor.Spin.update, and it uses Propellor.Protocol.req which reads from stdin. So, I think that putting
hSetEncoding stdin utf8in the update function may fix it for you.If so, the real fix will involve making propellor force utf8 on both sides of its protocol, because the spin might be run in some other locale too. (Or chainging to a binary protocol that doesn't suffer from encoding mismatch problems, if someone is ambitious!)
hSetEncoding stdin utf8inupdatedoesn't seem to work unfortunately, not sure why not.Since
updatereceives the changes to propellor's source code, it would have been running the old code at that point. You probably need to spin a second time to test your changes to that function.