There s something weird with the app bundle ```miniseph o se osquery #core

There’s something weird with the app bundle: ```mi...

seph

08/16/2021, 1:29 AM

There’s something weird with the app bundle:

Copy code

miniseph:o seph$ /usr/sbin/spctl -a -vvv  --ignore-cache opt/osquery/lib/osquery.app
opt/osquery/lib/osquery.app: code has no resources but signature indicates they must be present

seph

08/16/2021, 1:32 AM

This is status/error: -67056:

code has no resources but signature indicates they must be present. Most probably structural damage, which should be referred to the developer

(says https://eclecticlight.co/2019/06/28/a-primer-on-code-signing-errors/)

seph

08/16/2021, 1:35 AM

I’m guessing this is because we sign the binary, but not the .app

seph

08/16/2021, 1:53 AM

There’s so much tied together it’s hard for me to quickly resign and test this

seph

08/16/2021, 2:02 AM

Okay. I went through resigning. A pretty normal process, I used the same codesign commands that we do with the release file. but against the

.app

. The notarized it. Now it shows as signed and entitled. I’m less sure how to fix the build to handle this, I think we’re dumping raw stuff into package-data, which is then signed, and then packaged. And the packaging step converts it to a .app. But I think that ordering is probably wrong. Instead, we should be shoving the .app into the package data. Which can then be signed, and packaged into to the tar.gz and pkg. (I’d probably say the package-data should basically be identical to the tar.gz)

seph

08/16/2021, 2:05 AM

I even kinda understand stuff. With status quo, I can run osqueryd from contents, but I’m skeptical it has the entitlements, and it generally looks unsigned. But, I can move it outside the .app bundle, and then it starts looking pretty normal.

seph

08/16/2021, 2:05 AM

With my signing the .app. Things display as codesigned and notarized. But if I copy the binary out of the app, it complains about being invalid

seph

08/16/2021, 2:09 AM

Which, ultimately, probably means we need to view the app bundle as somewhat more real than we thought. I think we can distribute a bare, signed, osqueryd. But it may not be as useful.

seph

08/16/2021, 2:22 AM

If anyone wants to play, here’s a zip. This was built on CI, but signed on my local machine.

osquery.app.zip

seph

08/16/2021, 2:22 AM

https://osquery.slack.com/archives/C08VA3XQU/p1629080549035400?thread_ts=1629077347.033500&cid=C08VA3XQU

theopolis

08/16/2021, 2:43 AM

I am not sure I am following exactly, but it's late and my brain is most likely fried. Are we OK if we sign the binary, then the app bundle? I would expect that allows the binary to be moved out of the bundle and used as a normal signed binary.

theopolis

08/16/2021, 2:46 AM

But I see what you are saying about https://github.com/osquery/osquery-codesign/blob/master/.github/workflows/release-generator.yml#L417 needing to sign the app. Let me think through this and see if there's an easy path.

theopolis

08/16/2021, 2:50 AM

I don't see a downside to the package_data including the app bundle layout. That means less logic in osquery-packaging

alessandrogario

08/16/2021, 11:42 AM

Additional note to why package_data is done like that: since not all package formats require the same files, we used a neutral archive that contrains a 'control' folder with all the files required for all different formats

alessandrogario

08/16/2021, 11:43 AM

One of the major issues we had before is that we couldn't generate an .MSI file from a the ZIP release because the WiX data was missing. The opposite was also true, we couldn't start from the .MSI to generate Choco because we were missing the nupkg configuration files

alessandrogario

08/16/2021, 11:45 AM

I think it is fine to include .app as an additional format if there is no other way, but I think it makes sense to keep package_data neutral and see if we can fix this the correct way (and keep the standard procedure to sign while the package is being made)

sharvil

08/16/2021, 11:46 AM

Taking a look...

seph

08/16/2021, 1:28 PM

Are we OK if we sign the binary, then the app bundle?

From what I can tell, signing the app bundle overwrites the sig on the plain macho binary.

sharvil

08/16/2021, 1:55 PM

I think the

spctl

output is a red-herring of sorts (while not ideal)... since our app bundle is a minimal set of things required for it to work I took the

osquery-a08056b68ef64b23d163fafdfe6f0e1681c80a5c.pkg

from the test s3 bucket, installed it on a fresh macOS 11 VM, and things are working fine. It is picking up the entitlements and also the provisioning profile correctly, and I can see

es_process_events

flowing in

sharvil

08/16/2021, 1:55 PM

I am still digging a bit into this...

seph

08/16/2021, 1:58 PM

I could run the

osqueryd

from the the bundle, but all the signature verification tools I have say it’s a bad sig. Which makes me think we’re asking for trouble,

sharvil

08/16/2021, 2:00 PM

any other tool apart from spctl?

seph

08/16/2021, 2:01 PM

spctl, codesign, https://objective-see.com/products/whatsyoursign.html

👍 1

sharvil

08/16/2021, 2:01 PM

Copy code

root@osquerytests-Mac # codesign -vvv --strict --deep --display /opt/osquery/lib/osquery.app
Executable=/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
Identifier=osqueryd
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=311860 flags=0x10000(runtime) hashes=9737+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=870a7d77eabc2d12c410060aed030bbe940a81fc
CandidateCDHashFull sha256=870a7d77eabc2d12c410060aed030bbe940a81fc94241ab9824f21f9a96f7348
Hash choices=sha256
CMSDigest=870a7d77eabc2d12c410060aed030bbe940a81fc94241ab9824f21f9a96f7348
CMSDigestType=2
CDHash=870a7d77eabc2d12c410060aed030bbe940a81fc
Signature size=9021
Authority=Developer ID Application: OSQUERY A Series of LF Projects, LLC (3522FA9PXF)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Aug 15, 2021 at 5:52:53 PM
Info.plist=not bound
TeamIdentifier=3522FA9PXF
Runtime Version=11.1.0
Sealed Resources=none
Internal requirements count=1 size=168
root@osquerytests-Mac _CodeSignature #

seph

08/16/2021, 2:05 PM

Copy code

dover:osq5 seph$ codesign -vvv --strict --deep opt/osquery/lib/osquery.app
opt/osquery/lib/osquery.app: code has no resources but signature indicates they must be present

sharvil

08/16/2021, 2:06 PM

I see there is a new

_CodeSignature/CodeResources

file in the signed app bundle

sharvil

08/16/2021, 2:06 PM

Copy code

root@osquerytests-Mac Desktop # find ~/Desktop/osquery.app
/Users/osquerytest/Desktop/osquery.app
/Users/osquerytest/Desktop/osquery.app/Contents
/Users/osquerytest/Desktop/osquery.app/Contents/_CodeSignature
/Users/osquerytest/Desktop/osquery.app/Contents/_CodeSignature/CodeResources
/Users/osquerytest/Desktop/osquery.app/Contents/MacOS
/Users/osquerytest/Desktop/osquery.app/Contents/MacOS/osqueryd
/Users/osquerytest/Desktop/osquery.app/Contents/Resources
/Users/osquerytest/Desktop/osquery.app/Contents/Resources/osqueryctl
/Users/osquerytest/Desktop/osquery.app/Contents/embedded.provisionprofile
/Users/osquerytest/Desktop/osquery.app/Contents/Info.plist
/Users/osquerytest/Desktop/osquery.app/Contents/PkgInfo
root@osquerytests-Mac Desktop # find /opt/osquery/lib/osquery.app 
/opt/osquery/lib/osquery.app
/opt/osquery/lib/osquery.app/Contents
/opt/osquery/lib/osquery.app/Contents/MacOS
/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
/opt/osquery/lib/osquery.app/Contents/Resources
/opt/osquery/lib/osquery.app/Contents/Resources/osqueryctl
/opt/osquery/lib/osquery.app/Contents/embedded.provisionprofile
/opt/osquery/lib/osquery.app/Contents/Info.plist
/opt/osquery/lib/osquery.app/Contents/PkgInfo

sharvil

08/16/2021, 2:07 PM

Copy code

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "<http://www.apple.com/DTDs/PropertyList-1.0.dtd>">
<plist version="1.0">
<dict>
	<key>files</key>
	<dict>
		<key>Resources/osqueryctl</key>
		<data>
		cfmKsVPNeZNg+/+Phdj1bSeUOE4=
		</data>
	</dict>
	<key>files2</key>
	<dict>
		<key>Resources/osqueryctl</key>
		<dict>
			<key>hash2</key>
			<data>
			XK+14UzR5NpyldAfI3YrGoH2v0L1pzZLt/Hb6yBA5ko=
			</data>
		</dict>
		<key>embedded.provisionprofile</key>
		<dict>
			<key>hash2</key>
			<data>
			X1tITOEGCKY4oqba+4SFfq3vhZW3rGSe8gjNcIk9t1M=
			</data>
		</dict>
	</dict>
	<key>rules</key>
	<dict>
		<key>^Resources/</key>
		<true/>
		<key>^Resources/.*\.lproj/</key>
		<dict>
			<key>optional</key>
			<true/>
			<key>weight</key>
			<real>1000</real>
		</dict>
		<key>^Resources/.*\.lproj/locversion.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>1100</real>
		</dict>
		<key>^Resources/Base\.lproj/</key>
		<dict>
			<key>weight</key>
			<real>1010</real>
		</dict>
		<key>^version.plist$</key>
		<true/>
	</dict>
	<key>rules2</key>
	<dict>
		<key>.*\.dSYM($|/)</key>
		<dict>
			<key>weight</key>
			<real>11</real>
		</dict>
		<key>^(.*/)?\.DS_Store$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>2000</real>
		</dict>
		<key>^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/</key>
		<dict>
			<key>nested</key>
			<true/>
			<key>weight</key>
			<real>10</real>
		</dict>
		<key>^.*</key>
		<true/>
		<key>^Info\.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^PkgInfo$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^Resources/</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^Resources/.*\.lproj/</key>
		<dict>
			<key>optional</key>
			<true/>
			<key>weight</key>
			<real>1000</real>
		</dict>
		<key>^Resources/.*\.lproj/locversion.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>1100</real>
		</dict>
		<key>^Resources/Base\.lproj/</key>
		<dict>
			<key>weight</key>
			<real>1010</real>
		</dict>
		<key>^[^/]+$</key>
		<dict>
			<key>nested</key>
			<true/>
			<key>weight</key>
			<real>10</real>
		</dict>
		<key>^embedded\.provisionprofile$</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^version\.plist$</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
	</dict>
</dict>
</plist>

Which is a plist of all files and "hashes"

sharvil

08/16/2021, 2:14 PM

yep, you are right! we should sign the

.app

bundle too

seph

08/16/2021, 2:14 PM

AFAICT there’s no point to signging the macho binary prior.

sharvil

08/16/2021, 2:14 PM

okay, I am gonna put a PR to try things out

sharvil

08/16/2021, 2:15 PM

might result in some failed workflows and notifications while i try things out

sharvil

08/16/2021, 2:15 PM

yes, I think so too

seph

08/16/2021, 2:15 PM

I can review PRs intermittently.

seph

08/16/2021, 2:16 PM

But I think it would be clean to have the .app generated in the osquery package data phase

👍 2

alessandrogario

08/16/2021, 2:45 PM

reading again, i can see your point

alessandrogario

08/16/2021, 2:45 PM

makes sense to do it that way

alessandrogario

08/16/2021, 2:46 PM

i was initially assuming the .app to be a package itself but it's totally wrong as it's in fact a binary

seph

08/16/2021, 4:06 PM

Yeah. The .app is a special directory structure around some binaries. It’s sorta like a package, in you might not need any more.

sharvil

08/16/2021, 5:35 PM

PR for

osquery/osquery

https://github.com/osquery/osquery/pull/7263

sharvil

08/16/2021, 5:35 PM

working on updating packaging repo next

sharvil

08/16/2021, 6:42 PM

Packaging PR: https://github.com/osquery/osquery-packaging/pull/11

sharvil

08/16/2021, 6:42 PM

Codesign PR: https://github.com/osquery/osquery-codesign/pull/28

sharvil

08/16/2021, 6:44 PM

would like your code reviews, whenever you get a moment @seph @theopolis

3 Views

Open in Slack

Previous Next