# Utf8 issues with alpine linux

**URL:** <https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888>\
**Category:** Help & Support\
**Created:** [June 25, 2019, 12:42am UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888 "2019-06-25T00:42:49Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![steve-baldwin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/steve-baldwin/32/473_2.png) [@steve-baldwin](https://forum.crystal-lang.org/u/steve-baldwin)\
**Post date:** [June 25, 2019, 12:42am UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/1 "2019-06-25T00:42:49Z")

</div>

Hi,

My application runs just fine from my laptop (OSX). I’m trying to build a docker image to run my app based on alpine (edge) linux. The application (a http server) builds fine and runs but crashes when trying to process a http response:

```
Unhandled exception in spawn(name: d7eb4036): Invalid encoding: utf-8 (ArgumentError)
  from /usr/lib/crystal/core/iconv.cr:29:35 in 'initialize'
  from /usr/lib/crystal/core/iconv.cr:9:3 in 'new'
  from /usr/lib/crystal/core/io/encoding.cr:61:16 in 'initialize'
  from /usr/lib/crystal/core/io/encoding.cr:60:5 in 'new'
  from /usr/lib/crystal/core/io.cr:496:7 in 'decoder'
  from /usr/lib/crystal/core/io.cr:545:10 in 'gets_to_end'
  from /usr/lib/crystal/core/http/client/response.cr:81:15 in 'consume_body_io'
  from /usr/lib/crystal/core/http/client/response.cr:105:9 in 'from_io?'
  from /usr/lib/crystal/core/http/client.cr:562:5 in 'exec_internal_single'
  from /usr/lib/crystal/core/http/client.cr:548:5 in 'exec_internal'
  from /usr/lib/crystal/core/http/client.cr:544:5 in 'exec'
  from /usr/lib/crystal/core/http/client.cr:666:5 in 'exec'
  from /usr/lib/crystal/core/http/client.cr:370:3 in 'post'
:

```

I’m guessing I’m missing some libraries. Here’s the apk command from my Dockerfile:

RUN apk add --update --no-cache crystal shards musl-dev libressl-dev zlib-dev libpq sqlite-dev

I’m using alpine edge because I want the current version of crystal.

As an extra data point, if I build my docker image from crystallang/crystal rather than alpine:edge it runs without crashing.

Any suggestions?

Thanks,

Steve

---

<div class="post-metadata">

**Author:** ![steve-baldwin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/steve-baldwin/32/473_2.png) [@steve-baldwin](https://forum.crystal-lang.org/u/steve-baldwin)\
**Post date:** [June 25, 2019, 3:36am UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/2 "2019-06-25T03:36:55Z")

</div>

Another data point. I’ve built against alpine:latest (3.10) which now contains crystal 0.29 and see exactly the same issue.

---

<div class="post-metadata">

**Author:** ![steve-baldwin](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/steve-baldwin/32/473_2.png) [@steve-baldwin](https://forum.crystal-lang.org/u/steve-baldwin)\
**Post date:** [June 25, 2019, 9:06am UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/3 "2019-06-25T09:06:34Z")

</div>

I’ve managed to reproduce the issue with a simple test case.

The git repo is [here](https://github.com/horrendo/t-alpine).

Changing the first line of the `Dockerfile` to

`FROM alpine:3.9`

causes the error to go away. This builds against crystal 0.27.0.

Cheers,

Steve

---

<div class="post-metadata">

**Author:** ![ysbaddaden](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/ysbaddaden/32/2252_2.png) [@ysbaddaden](https://forum.crystal-lang.org/u/ysbaddaden)\
**Post date:** [June 27, 2019, 11:30am UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/4 "2019-06-27T11:30:06Z")

</div>

Something changed in musl-libc or alpine 3.10 related to iconv encodings. Maybe it needs an additional apk?

---

<div class="post-metadata">

**Author:** ![TheMasterCado](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/themastercado/32/480_2.png) [@TheMasterCado](https://forum.crystal-lang.org/u/TheMasterCado)\
**Post date:** [June 28, 2019, 1:54pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/5 "2019-06-28T13:54:00Z")

</div>

Same error here, trying to build a static executable from the latest alpine docker image, but http/client throws an exception about utf-8 encoding. Any idea about which apk would be missing ? Been trying to investigate the changes in musl-libc, but I found nothing.

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [June 28, 2019, 2:20pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/6 "2019-06-28T14:20:43Z")

</div>

Write a small program like this:

```crystal
from = "utf-8"
to = "utf-8"
Iconv.new(from, to)

```

if that fails, try with “UTF-8” and see if that works. If that works it means that `iconv` changed what they accept for valid encodings, maybe they dropped support for “utf-8” and they only accept “UTF-8”, I don’t know.

Could someone try this on their alpine docker image?

The fix might be to always send encodings uppercased.

---

<div class="post-metadata">

**Author:** ![TheMasterCado](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/themastercado/32/480_2.png) [@TheMasterCado](https://forum.crystal-lang.org/u/TheMasterCado)\
**Post date:** [June 28, 2019, 2:33pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/8 "2019-06-28T14:33:38Z")

</div>

So, no error with the exact code you wrote.

Crystal 0.29.0  
musl 1.1.22-r2

---

<div class="post-metadata">

**Author:** ![asterite](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/asterite/32/60_2.png) [@asterite](https://forum.crystal-lang.org/u/asterite)\
**Post date:** [June 28, 2019, 2:51pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/9 "2019-06-28T14:51:38Z")

</div>

I guess you’ll need to debug the application to find out what’s getting to `Iconv.new` and why it’s failing.

This is easy: you can modify `iconv.cr` directly in your Crystal distribution, or reopen it, copy the `initialize` method in your program and add debug info there (`puts`). Same goes with whatever you get from the http client.

It would be helpful to know which HTTP endpoint you are hitting too.

But all of this can be solved with debugging the code.

---

<div class="post-metadata">

**Author:** ![TheMasterCado](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/themastercado/32/480_2.png) [@TheMasterCado](https://forum.crystal-lang.org/u/TheMasterCado)\
**Post date:** [June 28, 2019, 3:07pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/10 "2019-06-28T15:07:19Z")

</div>

Good, will do that when I have time.

I’m hitting the bitbucket api ([api.bitbucket.org](http://api.bitbucket.org)) and using basic auth.

Thanks for the info

EDIT:

@asterite @steve-baldwin

While compiling the :musl flag isn’t on even if it’s compiling with it.

if you add `-Dmusl` when building it will set it manually. I don’t know why it’s not on or if it should be.

Hope this help you!

---

<div class="post-metadata">

**Author:** ![randummy](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@randummy](https://forum.crystal-lang.org/u/randummy)\
**Post date:** [July 31, 2019, 9:48pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/12 "2019-07-31T21:48:58Z")

</div>

I’ve been debugging this with

```
  def initialize(from : String, to : String, invalid : Symbol? = nil)
    original_from, original_to = from, to
    puts from
    puts to
    puts invalid

```

in /usr/lib/crystal/core/iconv.cr

This is what it’s showing:

```
utf-8
UTF-8
skip

```

Crash is reproducible with

```
from = "utf-8"
to = "UTF-8"
invalid = :skip
Iconv.new(from, to, invalid)
```

---

<div class="post-metadata">

**Author:** ![randummy](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@randummy](https://forum.crystal-lang.org/u/randummy)\
**Post date:** [July 31, 2019, 9:56pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/13 "2019-07-31T21:56:24Z")

</div>

Setting “@skip\_invalid = false” in /usr/lib/crystal/core/iconv.cr “fixes” the issue.

Why call iconv at all when trying to convert utf-8 to UTF-8? Only call it if lowercase(from) != lowercase(to).

Edit: Building with `-Dmusl` (`crystal build -Dmusl ...`) fixes the issue for me. See [https://github.com/crystal-lang/crystal/issues/7946](https://github.com/crystal-lang/crystal/issues/7946)

---

<div class="post-metadata">

**Author:** ![j8r](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.crystal-lang.org/j8r/32/111_2.png) [@j8r](https://forum.crystal-lang.org/u/j8r)\
**Post date:** [August 22, 2019, 4:00pm UTC](https://forum.crystal-lang.org/t/utf8-issues-with-alpine-linux/888/14 "2019-08-22T16:00:16Z")

</div>

I can reproduce it in Alpine 3.10, but not in Alpine Edge.  
We can consider it being fixed in the future stable release, 3.11.